MetroFi Receives Rubber Stamp
Posted by Caleb Phillips on 11 April 2007We just heard that Uptown Services has finished their report, and provided it to the city. In sum, their report gives the MetroFi network a passing grade, which has convinced the city to apparently simultaneously issue a certificate of acceptance to MetroFi based on the recommendation of project manager, Logan Kleier. Although we have not seen the full report from Uptown, we have a few comments on the rationale for acceptence. The bulk of the results from Uptown generally agree with our results. The results for network availibility, for instance, match closely with ours. The one clear exception is coverage. Mr. Kleier reports:
Uptown services data show that over 95% of outdoor coverage areas receieve a -79 dBm signal or greater. According to internal MetroFi data, -79 dBm is the minimum signal strength threshold necessary to connect to a MetroFi access point at 1 Mbps modulation rate. This exceeds the minimum RFP metric of 90% coverage in outdoor areas of the proof of concept.
There are a few problems with this statement. We were surprised to read the value -79 dBm, which was not documented in any RFP or contract that we are aware of. Further, the value of -79 dBm, without being qualified by a description of client hardware, or statistical demonstration that this correlates with the required 64 Kbps upstream and 64 Kbps downstream connection, is completely meaningless. The original RFP clearly indicates in Section 2.2.1.5, that the intention is to provide connections to lowest common denominator devices. At best, signal strength is half of the needed information to estimate connectivity coverage. Our methodology, in contrast, measured the ability to acheive a connection. Lastly, to use a value provided by MetroFi to test the MetroFi network, and to trust it blindly, is clearly not independent.
At this point, the MetroFi network has been rubber-stamped and will proceed forward. We are disappointed that the city chose to approve a network which has garnered much public skepticism immediately, and without an opportunity for peer-review or public comment. At best, this seems to indicate a substantial lack of due-dilligence. We look forward to studying the full report, and hope that it includes a clear description of Uptown’s testing methodologies and their collected data. We intend to finish our report, which is nearing completion, and will publish it here as soon as it’s complete.
April 12th, 2007 at 2:21 pm
An Open Letter to Logan Kleier
Dear Logan,
I’m writing to express my concern over the City’s recent sign-off on the MetroFi Proof of Concept deployment. Based on the only publicly released document from Uptown, there is little reason for the public to have faith in the City’s decision to issue the Certificate of Acceptance.
As someone who has worked with outdoor wireless network design and deployment for many years, initially as a volunteer with Personal Telco and later as the administrator for that group’s first grant, a wireless network along Mississippi Avenue; as a private contractor deploying networks between buildings in downtown Portland and most recently at Stephouse Networks, an ISP operating a number of wireless services in downtown Portland, St. Johns and the N. Marine Dr. area, I believe I can speak with some authority about the issues facing outdoor, urban wireless deployment, particularly those delivering service over the 2.4GHz band.
Throughput:
The Uptown Services Executive Summary, the only document available to the public from Uptown, begins by stating that 69 of 70 tested access points were shown to be capable of passing downstream traffic at 1Mbps and 70 of 70 were able to pass upstream traffic at 256 kbps. The opening figure for bandwidth capability is not indicative of performance over the 802.11g tier, where users experience it, but rather is an assessment of the performance of the 802.11a backhaul. Any client capable of connecting to an 802.11g AP ought to see these speeds between the client and AP.
Uptown goes on to state that their testing indicated that this level of performance was supported for 2 downstream clients, but only 1 upstream client, suggesting that the advertised performance is only available when a single user is on each AP. In short, the throughput assessment, as described in the Executive Summary, appears to ignore the presumable average load on the network, and instead presents a standard for success that virtually any functioning network should be able to meet.
Availability:
As many formal and informal usage tests have shown, when a reliable connection to the MetroFi network is found, it tends to remain reliable, as such, the 91% figure, as presented, is neither surprising nor impressive. It is entirely conceivable that a single device, placed in a location that had reliable coverage and connected continuously to the network, would have service availability of 91% or more. However, a single device connected in a location chosen for network reliability, is neither a rigorous nor a scientific means of testing average network performance.
The Uptown “drive testing”, like the throughput testing before it, presents success criteria that seem to ignore real-world use to the point of being meaningless. The ability of a single user to achieve the advertised performance is of little value if performance degrades with the second user on any given AP.
Of all the availability results, the most unsettling is the -79dBm figure, upon which network coverage has been judged. As anyone familiar with wireless networking can attest, a single signal strength reading is absolutely meaningless. All that this assessment means is that the MetroFi network can be “heard” at a certain level in 95% of the coverage area; it provides no insight whatsoever into connection availability. This is especially troubling, since MetroFi has repeatedly blamed end-users (and their hardware) for the inability to connect to the network. The inclusion of this statistic, is irresponsible and misleading without additional information about the strength or other technical assessment of the receiving equipment.
I can only presume that prior to issuing the Certificate of Acceptance, you received a more in-depth report from Uptown Services that clarifies some or all of my concerns regarding the testing methodology. It would behoove the City to release all or part of that report to demonstrate that the decision to sign-off on the MetroFi PoC was based on appropriate data and strong testing methodologies. The Executive Summary raises serious questions about this, especially in light of many assessments from frustrated users and the excellent research by Russell Senior and Caleb Phillips which suggest that usable coverage and performance are highly variable and MetroFi’s own statements regarding the probable need for additional hardware in order to connect.
Thank you for your time,
Michael Weinberg