Tange (2011): GNU Parallel - The Command-Line Power Tool, login: The USENIX Magazine, February 2011:42-47. When using GNU parallel for a publication please cite: configure | tail | parallel - j0 apt - file search AUTHOR if the lines you are looking for in bigfile all looks like: If you can convert your regexps into fixed strings do that. So if regexps.txt is 1% of your RAM, then it may be too big. Grep -F takes around 100 bytes of RAM and grep takes about 500 bytes of RAM per 1 byte of regexp. The normal grep -f regexps.txt bigfile works no matter the size of bigfile, but if regexps.txt is so big it cannot fit into memory, then you need to split this. The only way to know for certain is to test and measure. It is harder to see if disk I/O is the limiting factor, and depending on the disk system it may be faster or slower to parallelize. when running top), then RAM is a limiting factor.ĬPU is also easy to measure: If the grep takes >90% CPU in top, then the CPU is a limiting factor, and parallelization will speed this up. RAM is easy to measure: If the grep process takes up most of your free memory (e.g. There are 3 limiting factors: CPU, RAM, and disk I/O. grep filters out the large images, and parallel finally uses wget to fetch the images. parallel then calls wget to get each collection, which is passed to jq to extract the URLs of all images. jq is then used to extract the URLs of the collections. Wget is used to get the JSON for the search query. One of the pictures in each of these collection is large. The search query returns JSON containing URLs to JSON containing collections of pictures. Search NASA using their API to get JSON for images related to 'apollo 11' and has 'moon landing' in the description. jpg EXAMPLE: Download Apollo-11 images from NASA using jq cgi service = "SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0" layer = "LAYER=BlueMarble_ShadedRelief_Bathymetry" set = "STYLE=&TILEMATRIXSET=EPSG4326_500m&TILEMATRIX=5" tile = "TILEROW=.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |