User talk:Rillke/bigChunkedUpload.js

Screenshot of the script's main dialog

Help

Gist: Helps you to get done with individual, huge uploads.

This script is intended to be used by power users who want to overwrite or upload new files over a slow connection, in relation to the file size. It assumes its users know or can imagine what a chunk, a byte, the upload stash and an offset are and how to write wiki markup. Hint: The latter is what you enter if you prefer not to use the VisualEditor. At Wikimedia Commons, the most important template is {{Information}}, which you certainly want to use on every of your file description pages.

If you don't know, you can still try your luck but don't blame me for any unexpected incidents.

Activation link probably under the edit tab. Otherwise it's in the Tools box of the side bar

How to install

Add

importScript('User:Rillke/bigChunkedUpload.js');

to your common.js.

Global or foreign wiki installation

Note that the uploads always go to the wiki you are on! Cross-Wiki upload is not supported yet.

Add

mw.loader.load( 'https://commons.wikimedia.org/w/index.php?title=User:Rillke/bigChunkedUpload.js&action=raw&ctype=text/javascript' );

to your global.js.

Make sure you remove any imports of this script at Wikimedia Commons before enabling global import, i.e. check your common.js and vector.js (or monobook.js respectively) that you do not import the script there.

How to use

Overwriting an existing file

Overwrite and chunked upload links
  1. Choose the second option of the upload link of any file page:
    Upload a new version of this file (chunked upload)
  2. Fill out the reason field in the pop up and select the file to upload.

Creating a new file

  1. Navigate to File:DesiredFileName.ext. Please amend the file name before selecting a file to be useful.
  2. Press upload (chunked) from your "Tools" box in the sidebar, or from the drop-down menu next to the create (or edit) tab. The location of the link depends on your system and settings.
  3. Fill out the file description field in the pop up and then select the file to upload.

Compatibility

For a rough estimation c.f. http://caniuse.com/xhr2

Troubleshooting

  • In case the last chunk was uploaded and you see an error, please go to your UploadStash and try publishing from there (click here).
    • If publishing doesn't work, you might be running across this error. To workaround it, follow the instructions here.
  • Always select a reasonable chunk size. Uploading a 500 MiB video in 100 KiB chunks will likely fail due to server quotas while assembling the chunk. Uploading a 100 MiB video through a crappy connection with a chunk size of 20 MiB will likely be unsuccessful, too because none of the chunks will make it in one piece to the server.
  • There is currently a known issue with larger DJVUs and PDFs. To successfully upload them, you have to un-tick the checkbox use stash and async (recommended for large videos and photos). Don't do this generally as it makes the upload process more unreliable.
  • If larger chunks are told be uploaded 100% within no time and you encounter errors, it is most likely a proxy is taking all the data from your browser. Check your browser's and system's proxy settings then.
  • Report any other issues to my talk page.

See also

Category:Big Chunked Upload

Discussion

stashfailed

Hi, Please see phab:T200820#4826332. Thanks, Yann (talk) 11:18, 16 December 2018 (UTC)

I tried again without "stash and async" (as recommended above), and I got another error: phab:T212101. Yann (talk) 16:03, 16 December 2018 (UTC)

Uploads blocked on the first chunk

Hi, I have several examples of uploads blocked after the first chunk. No error message. Thanks, Yann (talk) 14:51, 21 March 2022 (UTC)

Again this happens all the time. Yann (talk) 08:54, 22 April 2022 (UTC)

FAILED: internal_api_error

Hi, I got 00030: FAILED: internal_api_error_UploadChunkFileException: [d187c3cd-06f9-4930-aa79-f4c504c6be32] Caught exception of type UploadChunkFileException while uploading a new version of File:The Collected Works of Mahatma Gandhi, vol. 52.pdf. Yann (talk) 10:06, 6 April 2022 (UTC)

Again now 00579: FAILED: internal_api_error_UploadChunkFileException: [155e92c0-67b3-4996-aaf9-4e2fc2a413b9] Caught exception of type UploadChunkFileException. Yann (talk) 11:16, 20 April 2022 (UTC)

Sever-Error 0

Hi, I got twice the message 18739: finalize/35> Sever-Error 0. Reason: error ... Still waiting for server to rebuild uploaded file and 20248: finalize/32> Sever-Error 0. Reason: error ... Still waiting for server to publish uploaded file after waiting a very log time, while uploading a new version of File:सम्पूर्ण गाँधी वांग्मय Sampurna Gandhi, vol. 56.pdf and File:सम्पूर्ण गाँधी वांग्मय Sampurna Gandhi, vol. 57.pdf. Yann (talk) 15:43, 28 April 2022 (UTC)

@Yann: I got the a similar-looking error, but with properly spelled "Server", without the hyphen, and with a lowercase second word. Sometimes, the tool has to wait a very long time to get sufficient info to write "Chunk uploaded". Along the way, it may display "upload is stuck" (about every 5 seconds until uploaded), "Connection seems to be okay.", "Waiting one more time...", "Re-sending this request.", or even "Server error 0 after uploading chunk:
Response: ". I have also gotten "Offset issue by Server detected. Attempting to fix automatically.", "Looks like this chunk was successfully transmitted but didn't receive a success message for it.Please have a look at the file after uploading.", and the preceding prefixed with "FAILED:". These messages can be explained as tolerating a good connection to a dodgy or overloaded system; it cannot all be explained by dodgy connection, machine, or script. Yes, I am in the process of trying to upload a colorized, 1437 × 1080, 60FPS, CQ24 version of Dixiana using a chunk size of 20480 KiB (20 MiB), with "use stash and async (recommended for large videos and photos)" checked. It is 4,938,565 KB on disk. If it succeeds, it will be named File:Dixiana 1930 (colorized, 1437 × 1080, 60FPS, CQ24).webm. I am uploading 242 chunks of 20.48 MiB each. Most of the chunks/parts upload successfully with a response in short order. I have seen 100 and 74 chunks/parts uploaded successfully the first try, but I have also seen failures. I have seen 242 chunks/parts uploaded successfully the third time, with the current result "finalize/242> Still waiting for server to rebuild uploaded file" displaying every 5 seconds or so since "09293: 242/242> Chunk uploaded" and "09293: finalize/242> Assembling chunks", and there have now been 25097-09293=15084s, or 263.4m, or 4.39h (4h23m24s) since the server started assembling the chunks. How long do I have to wait?   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 15:58, 11 January 2025 (UTC)
I waited until more than double the first start of assembly time of 9293 seconds (76000 and 34000) had passed, and had enough, so started again. I also started phab:T383554.   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 10:40, 13 January 2025 (UTC)
I also have gotten the following with "use stash and async (recommended for large videos and photos)" disabled as a test:

07550: 242/242> upload in progress Upload: 100%
07651: 242/242> Server error 500 after uploading chunk: Response: <!DOCTYPE html> <html lang="en" dir="ltr"> <meta charset="utf-8"> <title>Wikimedia Error</title> <style> * { margin: 0; padding: 0; } body { background: #fff; font: 15px/1.6 sans-serif; color: #333; } .content { margin: 7% auto 0; padding: 2em 1em 1em; max-width: 640px; } .footer { clear: both; margin-top: 14%; border-top: 1px solid #e5e5e5; background: #f9f9f9; padding: 2em 0; font-size: 0.8em; text-align: center; } img { float: left; margin:
07651: 242/242> upload in progress Upload: 98%

  — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 15:59, 13 January 2025 (UTC)
... and finally:
"11516: 242/242> upload in progress Upload: 98%" and "11536: FAILED: internal_api_error_DBQueryError: [a71060cb-bb92-4d28-8d32-70114068f9c3] Caught exception of type Wikimedia\Rdbms\DBQueryError".   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 17:49, 13 January 2025 (UTC)
For reference, the detailed error message for the dbquery error is:
[a71060cb-bb92-4d28-8d32-70114068f9c3] /w/api.php   Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transaction Function: UploadFromChunks::updateChunkStatus Query: UPDATE  `uploadstash` SET us_status = 'chunks',us_chunk_inx = 241,us_size = 5057090413 WHERE us_key = '1bi3kt156yw0.rbxhv5.122116.'
Its somewhat expected that disabling async will fail on files this size, so that's not too unexpected. Bawolff (talk) 07:46, 19 January 2025 (UTC)
@Bawolff: Thanks. New messages for key 1bikk4ud2p1k.nvqkqd.122116.:

07881: 199/242> Server error 0 after uploading chunk: Response:
07881: 199/242> Connection seems to be okay. Re-sending this request.
07881: 199/242> Connection seems to be okay. Re-sending this request. Upload: 99%
07906: 199/242> Offset issue by Server detected. Attempting to fix automatically.
07906: 199/242> Looks like this chunk was successfully transmitted but didn't receive a success message for it.Please have a look at the file after uploading.
07906: 199/242> Chunk uploaded
07906: FAILED: Looks like this chunk was successfully transmitted but didn't receive a success message for it.Please have a look at the file after uploading.

  — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 10:19, 19 January 2025 (UTC)
And another for key 1bilipzz0v14.hla65z.122116.:

01172: 27/242> Server error 0 after uploading chunk: Response:
01172: 27/242> Connection seems to be okay. Re-sending this request.
01172: 27/242> Connection seems to be okay. Re-sending this request. Upload: 99%
01195: 27/242> Offset issue by Server detected. Attempting to fix automatically.
01195: 27/242> Looks like this chunk was successfully transmitted but didn't receive a success message for it.Please have a look at the file after uploading.
01195: 27/242> Chunk uploaded
01195: FAILED: Looks like this chunk was successfully transmitted but didn't receive a success message for it.Please have a look at the file after uploading.

Now trying a smaller "Chunk size: 10244 KiB".
  — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 10:52, 19 January 2025 (UTC)

00135: FAILED: stashfailed: Internal error: Server failed to store temporary file.

Cannot overwrite File:Map of the city and county of New York - with the adjacent country - by David H. Burr. NYPL433800.tiff with rotated version via bigChunkedUpload. Error message: 00135: FAILED: stashfailed: Internal error: Server failed to store temporary file. What can I do? Sebastian Wallroth (talk) 19:41, 15 December 2023 (UTC)

@Sebastian Wallroth: I don't think this problem is related to the script. We have some phabricator tasks about the Upload Wizard encountering that error documented at COM:HD#Wizard upload not always working. You are welcome to comment at any of them.   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 01:00, 16 December 2023 (UTC)
Thank you. Sebastian Wallroth (talk) 05:27, 16 December 2023 (UTC)
@Sebastian Wallroth: You're welcome.   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 15:56, 16 December 2023 (UTC)
Category:Big Chunked Upload