Uhurulabs

New Firewall

17th April 20179th May 2017
by Frederick Mbuya

A few months ago my home firewall, (pfsense) died, was built in an old Dell Optiplex 755 that I picked up second hand for about 300,000TZS (150USD), it had served me perfectly for over 2 years so was not too upset when it died. I threw in a Zyxel UG20 firewall device that I have had hanging around, and I have not been happy, but more on that in another blog post. So today I decided to go through some of the junk I have around to see if I could not build another pfsense firewall, anyone who knows me knows I have lots of junk. I settled on an old HP chassis I have, major overkill as its 8GB and even found a SSD in it. This used to be my primary workstation, (after our office was broken into about 10 years ago and my very very very special machine was stolem 🙁 ), it then became my home server, and then about 2 years ago was retired, (figured I at the time I would repurpose it, but did not think it would take this long).

So yes its major overkill for a home firewall, but I will put a whole bunch of other networking tools on it….

13:09

So firewall is installed and in place, now for some quick iperf tests but figure it might be worth documenting, (as I always have to do a refresher, on what to expect)…

Network Core	Which is..	And means in theory the fastest you can transfer data is…
10Mbps	10 megabits per second	1.25 MB/s
100Mbps	100 megabits per second	12.5 MB/s
1000Mbps (1Gbps)	1000 megabits per second	125 MB/s

The iperf test was run between the new pfsense server and one of the pfesense boxes that is only 5 hops away:

------------------------------------------------------------
Client connecting to X.X.X.X, TCP port 5131
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  3] local Y.Y.Y.Y port 7131 connected with X.X.X.X port 5131
 ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  6.00 MBytes  10.1 Mbits/sec
[  3]  5.0-10.0 sec  6.00 MBytes  10.1 Mbits/sec
[  3]  0.0-10.0 sec  12.0 MBytes  10.0 Mbits/sec

A second test using www.speedtest.com, also show a max transfer of 8mb/s up and 10.1 down, so its safe to assume we are being capped at 10mb/s. Now though I would like to find out if that is a ISP cap, or there is a 10Mb physical link somewhere…

Blog

Trying to merge lots of big GeoTIFF’s

4th April 20172nd January 2019
by Frederick Mbuya

The idea was to merge into 1 geotiff 28 irregular shaped geotiffs which ranged in size from 1g up to 13g and making up a total of 113g. Why? Because I then needed to cut the resulting geotiff into multiple irregular shaped individual geotiff’s. This took a couple of days, and I was quite keen to come home today as this morning it was at 80%, you can only imagine my disappointment when I checked and found…

0...10...20...30...40...50...60...70...80...90..Traceback (most recent call last):
  File "/usr/bin/gdal_merge.py", line 540, in <module>
    sys.exit(main())
  File "/usr/bin/gdal_merge.py", line 526, in main
    fi.copy_into( t_fh, band, band, nodata )
  File "/usr/bin/gdal_merge.py", line 270, in copy_into
    nodata_arg )
File "/usr/bin/gdal_merge.py", line 63, in raster_copy
    nodata )
  File "/usr/bin/gdal_merge.py", line 105, in raster_copy_with_nodata
    nodata_test = Numeric.equal(data_src,nodata)
MemoryError

This as you can imagine was very frustrating, and I just assumed that the output was junk but figured what the hell, its created a 301G file lets see what it is. Started the process to load into Qgis, and since I figured it would take a while started this blog post. It seems though that output might be useful as qgis eventually loaded the file, and it actually looks like what I expected. So first things first I have set qgis to now save the file under a new name… Its going to take a while, currently at 12.2G and I actually expect it to end up bigger than the initial 301G

… a day or so later …

So it turns out the merge worked, I will revisit why the error later, but the file saved by qgis had the exact same size, and has the same result using gdalinfo. AND i have just done my first clip BUT it seems I made a mistake the command I used was

gdalwarp -dstnodata 0 \
-q -cutline shape_file_to_clip_to.shp -tr 0.05806 0.05806 \
-of GTiff input_file.tif \
clipped_file.tif

Which resulted in a file the same size of the input file, and the same dimentions, what I should have done, I think, is:

gdalwarp -dstnodata 0 \
-q -cutline shape_file_to_clip_to.shp -tr 0.05806 0.05806 \
-of GTiff input_file.tif \
clipped_file.tif

Which is what I am running now, the previous command took about 3 hours!

Blog

GeoTIFF Transparency

31st March 20172nd January 2019
by Frederick Mbuya

I have spent way too much time messing with this, and certainly too many CPU cycles processing only to get the wrong output. The problem…

“Two irregular shaped geotiffs that you need to merge into one and not have any stupid gaps”

My first attempts resulted in things like…

Not the desired output, but the fix was easy really, first fix the nodata value with

gdal_translate -a_nodata 0 \
-of GTiff \
file1.tif \
file1_nodata0.tif

gdal_translate -a_nodata 0 \
-of GTiff \
file2.tif \
file2_nodata0.tif

Then merge the two

gdal_merge.py -n 0 \
-a_nodata 0 \
file1_nodata0.tif file2_nodata0.tif \
-o merged_file

That results in a nicely merged GeoTIFF

Blog

Working with GeoTIFF’s from Pix4D

28th March 20172nd January 2019
by Frederick Mbuya

We have been doing a lot of work with drones over the past couple of years. Much of it proof of concept for the use of small, under 1Kg drones to capture aerial imagery as alternative to both manned aircraft and satellite, the premise being deploying small drones is much easier and cheaper than the alternative and that the acquired data would be of equal if not superior quality. For the most part we have used senseFly ebee drones, and done our post processing work in Pix4D Mapper, both being professional grade survey hardware and software. There are many possible outputs, but one of the main ones is a Orthophoto, according to Wikipedia

An orthophoto, orthophotograph or orthoimage is an aerial photograph or image geometrically corrected (“orthorectified”) such that the scale is uniform: the photo has the same lack of distortion as a map. Unlike an uncorrected aerial photograph, an orthophotograph can be used to measure true distances, because it is an accurate representation of the Earth’s surface, having been adjusted for topographic relief,^[1] lens distortion, and camera tilt.

The format of the orthophoto as output from Pix4D is GeoTiff and using gdalinfo we can extract the following info:

Driver: GTiff/GeoTIFF
Files: oysterbay.tif
Size is 34565, 37051
Coordinate System is:
PROJCS["WGS 84 / UTM zone 37S",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",39],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","32737"]]
Origin = (529846.625330000068061,9252232.345700001344085)
Pixel Size = (0.049060000000000,-0.049060000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=pix4dmapper
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  529846.625, 9252232.346) ( 39d16'12.33"E,  6d45'53.64"S)
Lower Left  (  529846.625, 9250414.624) ( 39d16'12.36"E,  6d46'52.83"S)
Upper Right (  531542.384, 9252232.346) ( 39d17' 7.57"E,  6d45'53.60"S)
Lower Right (  531542.384, 9250414.624) ( 39d17' 7.61"E,  6d46'52.80"S)
Center      (  530694.505, 9251323.485) ( 39d16'39.97"E,  6d46'23.22"S)
Band 1 Block=34565x1 Type=Byte, ColorInterp=Red
  NoData Value=-10000
Band 2 Block=34565x1 Type=Byte, ColorInterp=Green
  NoData Value=-10000
Band 3 Block=34565x1 Type=Byte, ColorInterp=Blue
  NoData Value=-10000
Band 4 Block=34565x1 Type=Byte, ColorInterp=Alpha
  NoData Value=-10000

This is for a sample 2 Gigabyte GeoTIFF. Now this might not sound too big but this is for an area of about 1.5sq/km. As you can imagine when you’re doing projects that are in the 100’s if not 1000’s of sq/km you will quickly need huge amounts of storage space. And that is not the only problem, as far as I know the most popular open source platform for hosting such data is currently a combination of geonode and geoserver, however after some experimentation I found that serving these images as is was simply not an option it was taking the geonode too long to render them, resulting in an awful user experience.

I looked into many options, and one of the best seemed to be mbtiles, however in the end it was not a suitable .. blog for another day…

After much research online it seemed like the best option was to first compress the GeoTIFF with gdal using gdal_translate. So far the best options I have been able to find are:

“-b 1 -b 2 -b 3”, which specifies that we only want the first three bands, if you refer to the gdalinfo output above you will see that for some reason Pix4D includes an alpha band. We don’t need it, I think! and its critical to only have the three bands or the PHOTOMETRIC flag won’t work. The next option is

“-a_nodata”, which sets the nodata, (transparency), value to 0 allowing the generated file to have transparency.

COMPRESS=JPEG is quite self-explanatory this is the instruction to gdal to use JPEG compression. This is better than the

PHOTOMETRIC=YCBCR, this changes the used color space from RGB to YCbCr, from what I have been able to find out. (thanks google), the eye is more sensitive to changes in luminance (Y, brightness) than to changes in chroma (Cb, Cr, color). Thus, it is possible to erase some chroma information while retaining image quality, thus allowing for better compression without any visible loss in image quality.

TILED=Yes again is quite self explanatory, it stores the image data in a tiled format which makes for a much better user experience when viewing the data using something like geonode or QGIS.

time gdal_translate \
-b 1 -b 2 -b 3 \
-a_nodata 0 \
-co COMPRESS=JPEG \
-co PHOTOMETRIC=YCBCR \
-co TILED=YES \
sample.tif \
sample_JPEG_YCBCR

After this the we have two files

151M Mar 28 17:54 oysterbay_JPEG_YCBCR.tif
2.0G Mar  2 07:49 oysterbay.tif

As you can see this has reduced our file from 2Gto 151M this is a 13 fold reduction is size, it took my workstation 39 seconds to do the compression, you can see details of my workstation here.

The next step is to add some zoom scales to the image, this will result in a slight increase in the file size but will result in a much better user experience when zooming into and out of the image.

time gdaladdo \
--config COMPRESS_OVERVIEW JPEG \
--config PHOTOMETRIC_OVERVIEW YCBCR \
--config INTERLEAVE_OVERVIEW PIXEL \
-r average oysterbay_JPEG_YCBCR.tif 2 4 8 16

This on my workstation took 28 seconds and added 50M to the file resulting in

220M Mar 28 18:29 oysterbay_JPEG_YCBCR.tif

And if we again check the file info using gdalinfo we get

Driver: GTiff/GeoTIFF
Files: oysterbay_JPEG_YCBCR.tif
Size is 34565, 37051
Coordinate System is:
PROJCS["WGS 84 / UTM zone 37S",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",39],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","32737"]]
Origin = (529846.625330000068061,9252232.345700001344085)
Pixel Size = (0.049060000000000,-0.049060000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=pix4dmapper
Image Structure Metadata:
  COMPRESSION=YCbCr JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (  529846.625, 9252232.346) ( 39d16'12.33"E,  6d45'53.64"S)
Lower Left  (  529846.625, 9250414.624) ( 39d16'12.36"E,  6d46'52.83"S)
Upper Right (  531542.384, 9252232.346) ( 39d17' 7.57"E,  6d45'53.60"S)
Lower Right (  531542.384, 9250414.624) ( 39d17' 7.61"E,  6d46'52.80"S)
Center      (  530694.505, 9251323.485) ( 39d16'39.97"E,  6d46'23.22"S)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  NoData Value=0
  Overviews: 17283x18526, 8642x9263, 4321x4632, 2161x2316
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  NoData Value=0
  Overviews: 17283x18526, 8642x9263, 4321x4632, 2161x2316
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  NoData Value=0
  Overviews: 17283x18526, 8642x9263, 4321x4632, 2161x2316

Note the GeoTIIFF is now compressed using YCbCr JPEG, and uses the YCbCr color space.
Bellow on the left is the section of the original Orthophoto and the right the compressed version, can you see a difference?

Here a section zoomed in even closer, 5 points to anyone who know’s what you’re looking at!

Blog

Tanzania Parliamentary Committee Visit COSTECH

23rd March 20172nd January 2019
by Frederick Mbuya

The Parliamentary Committee on Science and Technology visited COSTECH on the 22nd of March 2017. Uhurulabs took the opportunity to showcase some of its UAV’s and explain how we have been working very hard with COSTECH to explore the various ways that UAV’s can be of benefit to Tanzania.

At the same time in Buni (The COSTECH hosted Innovation space), Uhurulabs was training a group of young girls on how to make a good pitch.

Check out all the photos from the day…

Tanzania Parliamentary Committee Visit COSTECH / Google Photos

New Firewall

Share this:

Trying to merge lots of big GeoTIFF’s

Share this:

GeoTIFF Transparency

Share this:

Working with GeoTIFF’s from Pix4D

Share this:

Tanzania Parliamentary Committee Visit COSTECH

Share this: