April 2017 – Uhurulabs

17th April 20179th May 2017
by Frederick Mbuya

A few months ago my home firewall, (pfsense) died, was built in an old Dell Optiplex 755 that I picked up second hand for about 300,000TZS (150USD), it had served me perfectly for over 2 years so was not too upset when it died. I threw in a Zyxel UG20 firewall device that I have had hanging around, and I have not been happy, but more on that in another blog post. So today I decided to go through some of the junk I have around to see if I could not build another pfsense firewall, anyone who knows me knows I have lots of junk. I settled on an old HP chassis I have, major overkill as its 8GB and even found a SSD in it. This used to be my primary workstation, (after our office was broken into about 10 years ago and my very very very special machine was stolem 🙁 ), it then became my home server, and then about 2 years ago was retired, (figured I at the time I would repurpose it, but did not think it would take this long).

So yes its major overkill for a home firewall, but I will put a whole bunch of other networking tools on it….

13:09

So firewall is installed and in place, now for some quick iperf tests but figure it might be worth documenting, (as I always have to do a refresher, on what to expect)…

Network Core

Which is..

And means in theory the fastest you can transfer data is…

10Mbps

10 megabits per second

1.25 MB/s

100Mbps

100 megabits per second

12.5 MB/s

1000Mbps (1Gbps)

1000 megabits per second

125 MB/s

The iperf test was run between the new pfsense server and one of the pfesense boxes that is only 5 hops away:

------------------------------------------------------------ Client connecting to X.X.X.X, TCP port 5131 TCP window size: 65.0 KByte (default) ------------------------------------------------------------ [ 3] local Y.Y.Y.Y port 7131 connected with X.X.X.X port 5131 ID] Interval Transfer Bandwidth [ 3] 0.0- 5.0 sec 6.00 MBytes 10.1 Mbits/sec [ 3] 5.0-10.0 sec 6.00 MBytes 10.1 Mbits/sec [ 3] 0.0-10.0 sec 12.0 MBytes 10.0 Mbits/sec

A second test using www.speedtest.com, also show a max transfer of 8mb/s up and 10.1 down, so its safe to assume we are being capped at 10mb/s. Now though I would like to find out if that is a ISP cap, or there is a 10Mb physical link somewhere…

Blog

Trying to merge lots of big GeoTIFF’s

4th April 20172nd January 2019
by Frederick Mbuya

The idea was to merge into 1 geotiff 28 irregular shaped geotiffs which ranged in size from 1g up to 13g and making up a total of 113g. Why? Because I then needed to cut the resulting geotiff into multiple irregular shaped individual geotiff’s. This took a couple of days, and I was quite keen to come home today as this morning it was at 80%, you can only imagine my disappointment when I checked and found…

0...10...20...30...40...50...60...70...80...90..Traceback (most recent call last):
  File "/usr/bin/gdal_merge.py", line 540, in <module>
    sys.exit(main())
  File "/usr/bin/gdal_merge.py", line 526, in main
    fi.copy_into( t_fh, band, band, nodata )
  File "/usr/bin/gdal_merge.py", line 270, in copy_into
    nodata_arg )
File "/usr/bin/gdal_merge.py", line 63, in raster_copy
    nodata )
  File "/usr/bin/gdal_merge.py", line 105, in raster_copy_with_nodata
    nodata_test = Numeric.equal(data_src,nodata)
MemoryError

This as you can imagine was very frustrating, and I just assumed that the output was junk but figured what the hell, its created a 301G file lets see what it is. Started the process to load into Qgis, and since I figured it would take a while started this blog post. It seems though that output might be useful as qgis eventually loaded the file, and it actually looks like what I expected. So first things first I have set qgis to now save the file under a new name… Its going to take a while, currently at 12.2G and I actually expect it to end up bigger than the initial 301G

… a day or so later …

So it turns out the merge worked, I will revisit why the error later, but the file saved by qgis had the exact same size, and has the same result using gdalinfo. AND i have just done my first clip BUT it seems I made a mistake the command I used was

gdalwarp -dstnodata 0 \
-q -cutline shape_file_to_clip_to.shp -tr 0.05806 0.05806 \
-of GTiff input_file.tif \
clipped_file.tif

Which resulted in a file the same size of the input file, and the same dimentions, what I should have done, I think, is:

gdalwarp -dstnodata 0 \
-q -cutline shape_file_to_clip_to.shp -tr 0.05806 0.05806 \
-of GTiff input_file.tif \
clipped_file.tif

Which is what I am running now, the previous command took about 3 hours!

New Firewall

Share this:

Trying to merge lots of big GeoTIFF’s

Share this: