Benefits of opening data

I am a member of the GEO Data Sharing Working Group (DSWG). Via DSWG, we got a note from Mr. Avner Furshpan, Director of Climatology, Israel Meteorological Service (IMS), on IMS's experience about moving from selling data to providing free access to it. Many of the benefits cited here are well-known but still, this makes for a remarkable documentation of first-hand experience with moving toward "open" as a strategy. I am publishing the document below to make it visible to others who may benefit from this thinking.

March 14, 2016

02647016 Ref:

Free access to Israel Meteorological Service Data

At the beginning of 2012, the Israel Meteorological Service (IMS) started to provide free access to most of its real-time data and historical observations. The move marked a dramatic change in the IMS data policy. This article outlines the IMS experience, providing information for other NHMSs on the benefits and challenges of moving to a free data model.

Until 2012, most of IMS's operational budget (without salaries) was depended on external income rather than government funding. This included the budget required to fulfill international obligations such as WMO, maintain weather stations and deploy radiosonde equipment. To generate this income the IMS had to sell data and products, including tailor made forecasts and climatic analysis. Some problematic issues emerged from this funding model including:

• Private good took priority over public good. The focus of the IMS was on chasing possible clients and tenders and providing the services to paying customers, rather than on nationally important projects. For instance, the large number of queries from the private sector caused a big burden on the climate department personnel, leaving less time for activities such as the production of climate atlas, wind energy atlas and climate change analysis.
• The search for income resulted in new expenses and decreased efficiency as charging for data needed specific personnel from both the economical and meteorological sections.
• Most of the IMS income came from governmental agencies, thus there was not really additional flow of money to the government, but rather circulation of money from one pocket to another inside the government (which of course had its administrative cost).
• The competition between the IMS and the private sector was not really according to the free market rules, since the IMS, as part of the government, had to work according to a fixed price tariff, known to the private sector, which therefore had the ability to win any tender. Thus, the government artificially and inefficiently set the market price and did not benefit from this intervention.
• Governmental agencies sometimes didn't buy relevant data due to budget constraints, leading to a temptation to use low quality data available freely on the web instead.
• Some governmental agencies considered the price of the IMS data to be expensive. This and the restrictions placed on its publication or distribution, meant that alternative governmental networks were established, causing duplication and redundant governmental expenses. The high prices also dissuaded private companies from developing products or applications based on the IMS data. Such private companies didn't have the resources to establish appropriate alternative networks.
• Despite the discount offered to academic researchers, the high amount of data such users required meant it was not affordable to many researchers. This led them to search for alternative sources of data (often not quality controlled) or to use lower resolution products.

The issues described above produced a lot of antagonism towards the IMS and led to consideration of other funding models in relation to data dissemination.

The revolution came as a result of the growing demand for the IMS data by government ministries and their increasing complaints about paying for that data. This, coupled with the "Open Government" policy which was being implemented at that time to improve publishing of governmental information on the web, provided an opportunity for the IMS director to seize the moment. An agreement was reached with the Transport and Finance Ministries for a fixed yearly governmental budget in exchange for making all data accessible to the public, free of charge (a move which without additional governmental budget would have led to a major income loss for the IMS).

Under the new framework established by the IMS and the Ministry of Finance, the IMS now focuses on the provision of basic meteorological services (Public Goods) free of charge, while leaving the special meteorological services provision (tailor made products) to the private sector.

Some of the basic and public meteorological services that are provided by the IMS include:

• Forecasts and warnings published on the IMS website.
• Specific forecasts and warnings, as well as relevant climatological information, given to the aviation and marine sectors in line with international standards and requirements (added value products, on top of those required by international standards, are supplied by the private sector).
• Meteorological data from the IMS stations made freely available through a dedicated governmental website. The website includes near- real-time data from Automated Weather Stations and historical quality controlled climatological data. Private companies can freely use the data but are not allowed to sell this information. They can use it to develop special products which can be charged for.
• Regular and special climatological reports, climate atlases and climate change monitoring reports published freely on the IMS website.
• Products from NWP models which are run operationally by the IMS are available via the IMS web site.
• Meteorological information required by the various ministries is supplied free of charge, according to their needs. The IMS advises the government on any subject related to weather and climate and conduct research in order to improve the usability of meteorological information by the government.

Under the new operating model the private sector provides a range of other services including:

• Meteorological queries which require data processing (as already mentioned -free access to raw data is possible via the internet).
• Tailor-made, specific or elaborated forecasts.
• Advanced graphic displays of meteorological data (beyond those provided by the IMS).
• Meteorological counseling for the private sector, including a forecaster advice and consultation. (Counseling and briefing of government offices, governmental emergency units, aviation authorities and aviators is still provided by IMS).
• Provision and maintenance of meteorological equipment.

The implementation of the new framework brought many advantages to the IMS. There was a drastic reduction in tasks requiring significant time but little technical expertise, allowing increased focus on those activities which require meteorological skills. As a result, professional personnel can now focus on activities which are of national priority and the IMS has been able to promote important projects that were delayed or not carried before the change (e.g. wind energy atlas, climate atlas, climate change monitoring analysis).

Under the new framework, the IMS’s reputation as a service provider has been improved and the antagonism towards the IMS has ended. Cooperation across government has improved under the new model, including that with important users such as the emergency and rescue authorities.

Globally many NHMSs are looking to improve the availability of their meteorological and hydrological data. Advances in information technology and increasing demands from users for free and accessible data are increasing pressures on services all over the world. The WMO has, for many years, supported members in their efforts to exchange and published data for the mutual benefit of all services and users. The World Weather Watch and current activity connected to WIGOS, along with Resolutions 40 (Cg-XII) and 25 (Cg-XIII), provide just a few examples. Often, however, traditional funding models, based on the sale of such information, present barriers to opening up data.

Services considering a switch to open and free data face a range of technical and policy issues. One central issue is that the possible loss of such direct income, leads to concerns about services ability to maintain observation network and fund the crucial work of data processing and quality control. In this regard the experience of the IMS presents a useful case study.

In light of the experience the IMS has gained since the new move was launched in 2012, two important recommendations can be given to other NHMSs wishing to implement similar changes. Firstly, the platform for disseminating data should be clear and simple to use. It should also provide the ability to download large amount of data simultaneously. Secondly, the move to free data access can be gradual, for example, providing raw data freely without giving up incomes from forecasting services at the first stage.

In conclusion, the IMS example suggests that where agreements can be reached with governments, new funding frameworks based on free and open access can provide a wide range of benefits to the public and private sectors, as well as to the NHMSs themselves.

Avner Furshpan
Director of Climatology
Israel Meteorological Service

This article is published on behalf of the Region VI Working Group on Climate and Hydrology’s Task Team on Data Operations and Management.