OpenData has been a hot topic for the past few years, it means many different things to different people, the commercial sector see’s, (should see?), the potential to leverage OpenData for the creation of new products and services which they can sell to make money, e.g AirBnb and Uber both rely on access to other companies spatial data, (maps). The public service sector should be able to use the data to create better public services, and government decision makers should be able to use the data to make informed decisions.
Note the use of words like “use” and “leverage”, both imply the ability to access the data, and more specifically to be able to access the data in a Machine Readable format. As you can imagine the data we are talking about comes from many different sources, and will actually exist in repositories/databases all over the internet, maintained and contributed to by many different actors. The different user groups who need to use the data will most likely do so with their own specialized tools, be it ArcGIS, QGIS, STATA, R etc, so it is critical that their tools can talk directly to the data they need, thus the need for the data to be in a Machine Readable format.
One of the issues, especially in Tanzania, (and probably for many other regions of the world), is who/what is going to seed that initial collection of data AND who/what is going to make sure that the people who could potentially use the data are aware of it and can actually access and use it. Increasingly donor funded projects are facilitating the collection of huge datasets, the Zanzibar Mapping Initiative, is an excellent example of that, where the whole island was mapped using small scale drones. This for the most part by Zanzibari’s for Zanzibar, and the data has been made available as OpenData via the ZMI Geonode, and OpenArialMap. This in itself is a great win, especially as the approach taken was to train Zanzibari students and surveyors so if nothing else it has increased capacity. We here at Uhurulabs rely on our commercial contracts as they allow us to subsidize our public/innovation sector work and for our commercial work we rely on professional human resource, see our pilots, two of which are from Zanzibar and products of the ZMI project.
So in the Data collection has been seeded and done, and it is available as OpenData, how has it been used so far? The data has been used for things like building footprint digitization, we hope to provide a link to that soon. It has also been used by the Zanzibar Commision of Lands for land use and city planning, we also hope to provide links to information about that. Another very exciting use case has been in the Open AI Tanzania Challenge. which invited data scientists to develop feature detection algorithms that can automatically identify buildings and building types using high-resolution aerial imagery. It is our hope and expectation that the classifiers that were developed will be released as OpenSource and as such we have asked on in the Challenge Forum.
Another very exciting initiative is Ramani Huria…
“Ramani Huria is a community-based mapping project that began in Dar es Salaam, Tanzania, training university students and local community members to create highly accurate maps of the most flood-prone areas of the city. As the maps have taken shape – their benefits have multiplied and their potential magnified, now serving as foundational tools for development within all socio-economic spheres beyond flood resilience. The project is supported by funding from the UK Department for International Development through the Tanzania Urban Resilience Programme“
We were lucky enough to be involved in this project, most of our work involved collected data using drones. It always seemed that to some degree the drones stole away some of the attention from the _real_ work that was going on, on the ground. Everyone was very impressed with the quality, speed and accuracy that drones were able to capture high quality aerial images, but the real rich, actionable data came from the boots on the ground. The data includes things such as if a particular house has been flooded, and if so to what height, if a structure is public or private etc. What is really special about this data is its OpenData, the collection of which has been funded by public money, and done for the most part by specially trained university students and now available for free for public servants to make informed decisions, and business to create new services from.
In the maps below you will see the standard OpenStreetMap basemap, all the data from Ramani Huria was contributed to OpenStreetMap. However you will notice more detailed layers on top that give you more information and allow a deeper zoom level. You can then use the feature info tool (i), to get detailed information on the various assets, e.g clicking on a building will give you information such as if it is residential or not.
It is not however just donor lead projects that are releasing OpenData. Tanzania joined the Open Government Partnership, (OGP) in 2011.
Tanzania declared its intention to join OGP during the launching meeting in September 2011, one of six in Africa that qualified to be involved in the OGP.
Make the data available – in bulk and in a useful format. You may also wish to consider alternative ways of making it available such as via an API.
Make it discoverable – post on the web and perhaps organize a central catalog to list your open datasets.
Note the words in RED, again these words get thrown around quite a lot but what do they mean in this context? 99% of the people who use an API will never know they are using it, it simply means that the data is stored in such a way that the software YOU use to get your job done can access it, without you needing to know the technical details of HOW its being accessed. An example of this is when you go to google maps and search for a location, and then tell the app to give you directions to there, you don’t need to know the technical details of HOW that is happening, or how it know’s that there is high traffic on a certain route. You simply want to enter a location and click a button, and it is the fact that many systems make their data available via an API, that allows it to be possible.
What does discoverable,mean in this context? Simply that a user or system is able to find the data when it needs it. This can be as simple as a search box on a website, or more complex e.g system that pushes you information based on your location and previously identified preferences.
Which now actually brings us the main point behind this post, yes it is true that Tanzania has made great strides in its ability to collect, valuable geospatial and other data. It has also made good progress in its ability to then make that collected data Open. Where I think it is now struggling is in the ability to store, manage and reliably make that data available in easy, useful, discoverable systems using things such as API’s and well crafted interfaces that can be accessed by all stakeholders. This however is a struggle that is not Tanzania’s alone, the whole world is seeing an explosion in the availability of data, it is coming from all kinds of sources, drones, new low orbiting satellites, weather stations, from government, private sector, individual citizens etc. At the same time there is an increasing awareness that good decision making comes from the utilization of good data. Last month we were invited to speak at the Smart Land Administration forum in Finland, where the topic of the day was Spatial Data Infrastructures, (SDI),
A spatial data infrastructure (SDI) is a data infrastructure implementing a framework of geographic data, metadata, users and tools that are interactively connected in order to use spatial data in an efficient and flexible way. Another definition is “the technology, policies, standards, human resources, and related activities necessary to acquire, process, distribute, use, maintain, and preserve spatial data”.[1]
A further definition is given in Kuhn (2005):[2] “An SDI is a coordinated series of agreements on technology standards, institutional arrangements, and policies that enable the discovery and use of geospatial information by users and for purposes other than those it was created for.”
Note that from the above definition it is clear that the weight is on the institutional policies, standards, and agreements which need to exist. Without good strong institutional policies, standards and agreements an SDI or anything like an SDI cannot exist and as such the public will NEVER realize the true potential of the Data revolution. Data will continue to be collected as the market understands it has value, but rather than a tool to liberate the general citizen, it will be a tool to control and oppress.
Keep going to the government websites I mentioned above, use the data, and when you have questions about it ask the relevant body which is very often NBS, the reality is for the most part people in Government are their and willing to help. It is only by the use of such data do people start to understand its value and are motivated to make sure it is kept up to date and available. If you find a website or service to be down then inform the owner.
At Uhurulabs we are committed to servicing and helping anyone who wants to use data and technology for the betterment of Tanzania and its people, as such we are committed to maintaining a number of services, the first of which is now live and is a Geonode that is available for anyone to host geospatial data, and for anyone who wants to access the data.
https://geonode.uhurulabs.org
Our resources are limited and as such we will try to scale the service according to demand, if you experience any problems please let us know by emailing to geonode@uhurulabs.org. For the most part what we will be trying to do is aggregate datasets that we find in other places, especially
data that is not available via API
data that we believe is at risk of being lost
data we have collected ourselves.
The second is one which came to mind while writing this article, and it will be a page that monitors the various websites and services that provide access to data on Tanzania.
The Revolutionary Government of Zanzibar (RGoZ) with the support of the World Bank has been developing the Open Data for Resilience Initiative (OpenDRI) with the aim of supporting evidence-based and innovative solutions to better plan, mitigate, and prepare for natural disasters.
Zanzibar is part of the Southwest Indian Ocean Risk Assessment and Financing Initiative (SWIO RAFI) which seeks to address high vulnerability of the Southwest Indian Ocean Island States to disaster losses from catastrophes such as cyclones, floods, earthquakes and tsunamis. These threats are exacerbated by the effects of climate change, a growing population and increased economic impacts.
The Zanzibar Mapping Initiative (ZMI) is a cooperative project between the Tanzania Commission for Science and Technology (COSTECH) and the Revolutionary Government of Zanzibar (RGoZ). COSTECH agreed to provide its in-house expertise and drones to support the creation of a new aerial base map of Zanzibar for the Zanzibar Commission for Lands (COLA).
UAV data has been collected over a large area +1,500 sq/km. The UAV’s used were small and as such the area was divided into 200+ zones each of which were then processed into individual Geotiffs. The data was collected without absolute accuracy and as so although the data within a given zone is relatively accurate there are varying degrees of edge matching issues when attempting to put all 200+ zones together.
Attempted solution…
A post processing process that would attempt to take individual zones and automatically do adaptive filtering of zones and then attempt to match using edge matching.
Test Data…
A 27 Zone section was selected as the same dataset.
Preliminary Result…
3km x 3km Zone.
pre
post
at a glance it looks good, so lets look a little closer…
Okay so I am finally writing this post. My initial target was to have it done by the end of January, but that slipped so next target was mid Feb, and that has slipped but better late than never!
Last year I together with Stephen Mather of OpenDroneMap were privileged to be invited to Kerala, India by International Centre for Free and Open Source Software (ICFOSS) to run a workshop on open source solutions for processing drone imagery and open hardware drone solutions. Checkout Stephen’s blog post here. As usual I was nervous, although I have been diving deep into the topics the past few years, I still get nervous when I am going to do a presentation or workshop.
Was a little hard leaving home, I don’t like leaving my girls for more than a couple of days, and even though this was only going to be 5 days, India felt like a long way away.
I left very early from home, but as usual the traffic in Dar es Salaam is totally unpredictably. By about half way it was clear I was not going to make it, so I started considering my options. Only one really, get a Bodaboda, or rather two Bodaboda’s, one for me and one for my luggage.
IMG_20180115_090923
IMG_20180115_090917
IMG_20180115_090727
Got to the airport in time and was able to board the flight to first Dubai and then India.
Arrived in India quite late at night around 4am, as usual I was a little concerned about getting through customs with all the gear but it was no problem at all, even with the big ebee case(link), and the tons of drone bits in my bags. Once outside the airport the person who was there to collect me seemed to recognize me immediately, (maybe he was told to look for the black guy with dreads!). He took me straight to the guest house which was on the Trivandrum Technopark campus, and within 20 mins I was in my room and about to get some rest, (by then it was 5am), when there was a knock on the door! I opened and there was a guy standing, telling me that they are ready to leave in 30 mins…
Although the organization that had invited us was based in Trivandrum Technopark, India’s Largest IT Park, the actual workshop was to be held at Vagamon some 190 km a hill station at least 5 hours drive.I have to admit I was a little rough saying “NO I have to get some rest”, he went away for a while and then came back to let me know it was okay, I would go later with another group! Nice, I could get some sleep.
Woke up around 10am, and of-course top priority was INTERNET! Unfortunately the guest house did not have internet, which seemed kinda strange as it was on a science park, so went off wondering to see if I could get a sim card. After wondering around for about an hour and talking to a number of people it seemed that you need an Indian ID to get a sim card, not so sure that was true, but figured I ought to get back to the guest house in-case my host was looking for me. Lucky I did as someone from my host turned up a few minutes later and we headed together to the ICFOSS offices.
I spent a couple of hours at the ICFOSS office, got online, checked in with my family back home and checked my emails etc. Then headed back to the guest house because the plan was to drive up into the mountains at around 17:00.
The drive was great, a little long, but it was so nice to see some of the Kerala countryside, arrived pretty late partly because we had stopped along the way to eat, FYI what people in Kerala consider junk food, many would consider a healthy snack!
DSC_0035
IMG_20180117_173947
DCIM100MEDIADJI_0417.JPG
IMG_20180118_130740
Once at the place we were spending the night Stephen and I spent some time planning the following day and then everyone got some much needed rest.
In the morning we headed to the place where we were going to hold the classroom part of the workshop. Before the rest of the attendees showed up we had some time to work on the Ranger, ahead of its maiden flight that was planned for the afternoon. We had good attendance, Arun M the lead from ICFOSS together with the core drone team, Deepthi, Vaisakh, and Asishwith, plus other ICFOSS staff, members of the Kerala electricity board, Veenus and Manoj from 1074 Vectors and others. All in all we had people from different sectors of government, startups and some more mature private sector companies.
IMG_20180117_100809
IMG_20180117_093523
IMG_20180117_105155
Stephen started off the workshop with introductions and then it was over to me to talk about some of the work we have been doing in Tanzania. For the most part I covered how drones had helped us with the mapping of several flood prone areas of Dar es Salaam in the Ramani Huria Project, and how the biggest project yet the mapping of Zanzibar with low cost drones, had gone on to win awards and really helped validate all the work we were doing. It was great to have representatives not only from government, but also from the private sector, all keen and interested in how drones could help their particular needs. I left them with some thoughts about how to engage with the relevant government stakeholders to ensure that not only were they building technical expertise, but also ensuring a regulatory framework that would encourage innovation and be supportive of local businesses. The afternoon was supposed to be a field session were the group would actually see some drones in action. One of the guys from ICFOSS and I went hunting for a good place for take-off and landing.
We were lucky and found a great spot, so we started with out field setup, we actually had quite a few drones with us…
Hobby King Duet. Small fixed wing, basically a park flier that can take a beating and keep flying making it perfect for training.
SenseFly Ebee, a great platform even if its proprietary.
The aim for the afternoon session was to demonstration how both a multi-rotor, (3DR SOLO), and a fixed wing (Ebee), can be used for mapping highlighting the difference and when each is most appropriate. Best case we were hoping to do some actual mapping with both platforms so that we could have some data that could be used for the OpenDroneMap processing demonstrations. Everything went perfectly, both the ebee and the 3dr solo were used to gather some data, and we even had some time to play around with the Mavic and really show the group how far consumer drones had come.
DCIM100MEDIADJI_0427.JPG
DCIM100MEDIADJI_0419.JPG
DCIM100MEDIADJI_0425.JPG
DCIM100MEDIADJI_0428.JPG
DCIM100MEDIADJI_0424.JPG
DSC_0040
DSC_0045
DSC_0039
DSC_0051
You may be asking yourself why we had a SenseFly Ebee at a workshop that was about OpenSource and OpenHardware? As it stands I still heavily recommend someone who is exploring the use of UAVs for mapping and other survey activities to get an Ebee IF their budget allows. I say this because the Ebee is a platform that just works, it comes with both the flight planning software, processing software, (Pix4D) and an excellent support infrastructure that is critical when you are getting into this field. Things will go wrong, and it’s nice in the beginning to minimize the things that could go wrong, and to have an organization that provides excellent support available, should you need it. There are many times when I have been out in the field doing a client survey and I have had to call SenseFly support to help get on some technicality or the other. Another reason why I recommend the ebee especially in countries where regulations are still in their infancy if they exist at all, is because of the extremely low risk the low weight of the ebee poses to people or property in the case of an accident, although I would not advise it there is even a video of a Danish firm doing a “test”.
That evening we all huddled together around the table planning for the next day. I processed the data we had collected during the day, Stephen and Deepthi planned the flights they expected to do the following day, and even decided to use the elevation data from my flights to plan other flights what we would do the following day. This was a great opportunity to show how valuable data generated from a quick drone survey could be, after just 2 hours in the field we already had elevation data that was more accurate than we could get from Google.
The following day we planned to do some quick flights in the morning and then head back to the city so that on the last day we could do some classroom data processing work. The flights went almost too perfectly, in my opinion you have not flown till you have crashed. The last flight of the 3DR Solo was done by Deepthi, and unfortunately, (or fortunately), there was a small issue during landing which broke 2 props, no huge issue and props are “consumables” anyway! This was also the last chance to test the , the area was not perfect, but we gave it a try anyway, Stephen on the controls, (the only one with real manual fixed wing experience), fact is though the runway was just not long enough and he had to abort at the last moment, one broken prop but the UAV still in perfect condition.
DSC_0052
DSC_0053
DSC_0054
DSC_0087
DSC_0086
DSC_0085
DSC_0084
DSC_0083
DSC_0053
Friday was the last day of the workshop held at the ICFOSS offices. Usually such workshops start with lots of time being wasted installing software and actually getting the environment ready. Not at ICFOSS, they had everything ready, a server already setup with OpenDroneMap, and a room setup with laptops for the participants. I wish every workshop I was involved with went this well. I won’t go into detail but some highlights were:
1) Everyone was able to connect to the ODM server and create their account without issue.
2) Everyone after a little bit of instruction was able to upload their data-set, and get the processing done.
But that was not the most impressive thing, while Stephen was driving the workshop I was preparing a 3D model of the work that I had made from the data using Pix4D. I have to admit was I was quite looking forward to showing it off, until I turned around and saw something very similar on one of the participants screen. I wandered over to ask what it was and it turns out, it was the 3D mesh from the work he had just done with OpenDroneMap, being rendered on windows using meshlab. I was humbled, excited and impressed all at once.
We closed the workshop, with clear plans on how we would like to move forward, and exciting possibilities of future collaboration, and a very strong feeling that there is so much we can learn from each-other. That evening we had a great tour of Kerala with Arun and Jucar, which ended with a great meal on the beach. All in all I really enjoyed this trip, and looking forward to my next, (hopefully with my family this time), and to the great collaborations that I hope will come from it between Uhurulabs, OpenDroneMap and ICFOSS.
Sustainia just launched their 4th Global Opportunity Report together with DNV GL and the United Nations Global Compact. Over the years, the Global Opportunity Reports have proved that no challenge or risk is too big for business to tackle, and that there is market potential in every one of the 17 Sustainable Development Goals, from smart farming eliminating hunger to solar micro-grids providing clean energy.
All market opportunities accumulated in the Global Opportunity Reports are now available on our innovation hub Global Opportunity Explorer for everyone to explore. But which market has the biggest positive impact on people, planet and profit?
So for a couple of days last week I went back to school. A few of us from africanDRONE, namely Unequal Scenes, Microdrone, African Defence Review and of course Uhurulabs, were sponsored by ICFJ to attend a three day intensive Drone Journalism School at the University of Oregon, in Portland.
The conference was organized by Google News Labs, the University of Nebraska School of Journalism and Poynter. It was a very interesting experience as for the most part my work with drones has been for data collection, survey etc, while most in the room were focusing on using drones for video and photography. One thing we all did have in common however was an overwhelming understanding that the rapid commercialization of consumer drones has changed the realm of what’s possible and by whom in ways we could not have thought possible five years ago.
A main focus of the conference was the training for the American FAA Regulations Part 107, which are the new rules for non-hobbyist small unmanned aircraft (UAS) operations in the USA. The conference had several great tutors who took everyone through all the aspects of the regulations in preparation for the attendees to take the exam.
Regulations are the topic of the day, and of great concern to anyone who wishes to operate UAS commercially. All over the world the various regulatory bodies are struggling to create rules and an environment that allow for the safe and fair usage of our skies. A colleague from africanDRONE has said that he thinks the American approach is “pragmatic and economically advantageous“, this of-course is relative to the South African regulations where it costs up-to 10,000$ to be fully registered and certified to operate UAS commercially. Contrast that with Tanzania where currently there is no payment required for the registration of UAS, there are only procedures you are required to follow with the Tanzanian Civil Aviation Authority (TCAA), and the Ministry of Defence (MoD). I personally have done this a number of times and have found the process straight forward. The one thing I will mention is that until now you have been required to do this for every flight, which can become a burden. I am currently in the process of getting a more permissive permit, that perhaps will only require me to log a flight plan whenever I want to fly, fingers crossed on that!
Lots of people know me as the Drone guy, what many don’t know is that I don’t think I am actually a very good drone pilot, I almost never fly for photography or video and much of my work is automated, where all flights are planned and programmed in the office. I rarely get my hands on the latest consumer drones and even when my friends offer me to try theirs I say no, as I don’t want to crash a $6,000 Inspire! Luckily for me one of the conference sponsors DJI, had brought a number of their latest drones and took us all out for some test flights. The inspire 2 is awesome, but would be a waste in my hands, happy though that I had a chance to finally fly it!
Although I have been using KVM, (Linux Based Kernel Virtualization), on my own infrastructure for some time, it is only recently, (past 4 years), that I have comfortable enough with it to use it in production for my clients, especially when other admins would be involved with administration of the systems. A choice I often made was to use Citrix XEN, as it…
Has a commercially supported version should my client decide to stop using me.
Has a fully free and open version which for the most part is not cobbled, at least it has all the core functionality that my clients need.
One thing that always frustrated me about it was the fact I needed to use a Windows machine to use access its management interface, yes there is a commmunity driven linux version but never worked quite well.
Anyway one of the clients where I had used Citrix XEN, is now ready for an upgrade and I found myself in a situation where I needed to move virtual machines from XEN to KVM. It took quite a bit of digging and experimenting but this is what I came up with.
First of all you need to gather three pieces of info:
NETWORK-UUID is eb57b4d8-7656-c607-b1cc-93cfd8766afe
We now need to tell the XEN Server to put the machine in “transfer” mode, NOTE that this will make the machine unavailable for the duration of the export process.
Now we have the URL: http://47b5ebf70160c752:f2b25a5854c12b76@192.168.13.88:80/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88, we can now use curl to get the image
I have been noticing with more and more concern the increasing amount of what looks to me as VERY young police officers on the streets carry what look to me like AF-47, (I don’t know much about guns!!!). When I have seen what feel to me like “gangs” of them on street corners I have honestly asked myself if I feel more or less comfortable. Fact is they make me nervous as hell, and part of me has been comforted by the fact that I figured that probably most of the guns are none functional or not loaded. I came across a video yesterday that confirmed that I should be very afraid, AND that my hope that the guns are not loaded or not working was crushed.. indeed they are loaded, they do work, and unfortunately looks like the police are somewhat lacking in their training.
A few months ago my home firewall, (pfsense) died, was built in an old Dell Optiplex 755 that I picked up second hand for about 300,000TZS (150USD), it had served me perfectly for over 2 years so was not too upset when it died. I threw in a Zyxel UG20 firewall device that I have had hanging around, and I have not been happy, but more on that in another blog post. So today I decided to go through some of the junk I have around to see if I could not build another pfsense firewall, anyone who knows me knows I have lots of junk. I settled on an old HP chassis I have, major overkill as its 8GB and even found a SSD in it. This used to be my primary workstation, (after our office was broken into about 10 years ago and my very very very special machine was stolem 🙁 ), it then became my home server, and then about 2 years ago was retired, (figured I at the time I would repurpose it, but did not think it would take this long).
So yes its major overkill for a home firewall, but I will put a whole bunch of other networking tools on it….
13:09
So firewall is installed and in place, now for some quick iperf tests but figure it might be worth documenting, (as I always have to do a refresher, on what to expect)…
Network Core
Which is..
And means in theory the fastest you can transfer data is…
10Mbps
10 megabits per second
1.25 MB/s
100Mbps
100 megabits per second
12.5 MB/s
1000Mbps (1Gbps)
1000 megabits per second
125 MB/s
The iperf test was run between the new pfsense server and one of the pfesense boxes that is only 5 hops away:
------------------------------------------------------------Client connecting to X.X.X.X, TCP port 5131TCP window size: 65.0 KByte (default)------------------------------------------------------------[ 3] local Y.Y.Y.Y port 7131 connected with X.X.X.X port 5131 ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 6.00 MBytes 10.1 Mbits/sec
[ 3] 5.0-10.0 sec 6.00 MBytes 10.1 Mbits/sec
[ 3] 0.0-10.0 sec 12.0 MBytes 10.0 Mbits/sec
A second test using www.speedtest.com, also show a max transfer of 8mb/s up and 10.1 down, so its safe to assume we are being capped at 10mb/s. Now though I would like to find out if that is a ISP cap, or there is a 10Mb physical link somewhere…
Blog
