HomeMy WebLinkAboutStaff Report 2309-2082CITY OF PALO ALTO
Finance Committee
Regular Meeting
Tuesday, November 07, 2023
Agenda Item
2.Discussion and Update of the Palo Alto Fiber Expansion Plan and Construction Alignment
with Electric Grid Modernization. CEQA Status – Not a project.
Finance Committee
Staff Report
From: City Manager
Report Type: ACTION ITEMS
Lead Department: Utilities
Meeting Date: November 7, 2023
Staff Report:2309-2082
TITLE
Discussion and Update of the Palo Alto Fiber Expansion Plan and Construction Alignment with
Electric Grid Modernization. CEQA Status – Not a project.
RECOMMENDATION
This memorandum and presentation are for discussion purposes only; no action is requested at
this time.
EXECUTIVE SUMMARY
On December 19, 2022, City Council directed staff to proceed with the Fiber Expansion Plan to
implement the Fiber Rebuild project and Phase 1 of the Fiber-to-the-Premises (FTTP) project. In
Phase 1, FTTP would be built out in selected areas of the city, and expanded gradually (Staff
Report ID # 14800)1. Construction of the fiber backbone in the Fiber Rebuild project and last mile
infrastructure to provide FTTP broadband internet to the community in the FTTP project will be
a significant undertaking for the City. Before construction commences, staff needs to address the
following to control project costs, minimize construction impacts to the community, and prevent
major delays:
✓Align fiber and grid modernization projects - Engineering make-ready work and
construction for the grid modernization project will overlap with projects in the Fiber
Expansion Plan. Staff analyzed the projects to identify opportunities for alignment. Staff
is deploying a pilot to help inform on how alignment efforts impact costs, reductions to
community disruptions, and project timelines.
✓California Environmental Quality Act (CEQA) - The City must analyze and evaluate the
potential impacts of the project on various environmental factors and identify whether
those impacts can be mitigated.
1 https://www.cityofpaloalto.org/files/assets/public/agendas-minutes-reports/agendas-minutes/city-council-
agendas-minutes/2022/20221219/20221219pccsm-amendedtime.pdf
✓Joint Poles - The existing joint pole agreement with AT&T requires coordination between
the City and AT&T to relocate 3rd party telecom equipment on utility poles and provide
space for new fiber attachments, while remaining in compliance with California Public
Utilities Commission General Order 95 (GO 95) requirements for overhead electric line
construction to insure electric utility service and secure personnel safety.
✓Contract amendments – Staff is evaluating the utilization of existing construction and
engineering design contracts to expedite design and construction for both FTTP and grid
modernization for the pilot. Existing contractors have the technical expertise and
familiarity with the City’s construction standards, so construction may begin as soon as
the power engineering design is complete.
BACKGROUND
City Council considered three courses of action for a City-owned FTTP service and directed staff
to proceed with the phased build approach. Under this approach, FTTP will be built out in
selected areas of the city using $34 million from the Fiber Fund and $13 million from the Electric
Fund, and the project will expand gradually from there to eliminate the need for debt financing.
Building the fiber backbone and last-mile infrastructure to provide FTTP broadband internet to
the community will require significant planning, coordination, communication, and construction
over the next 18 – 24 months.
On May 1, 2023, City Council approved Amendment Number 4 to Contract Number C20176363
with Magellan for Fiber program management, organizational change management, network
operations and technical support, and utility pole electric make-ready engineering (Staff Report
#2303-12152). Although the phased build approach to FTTP will decrease the City’s financial risk
and increase Council’s control over the velocity of the buildout, the City does not have the in-
house staffing to fully pursue the Fiber Expansion Plan and efforts to coordinate with
Electrification Grid Modernization. These programs require significant staffing and specialized
skill sets over the next few years. While it is economically prudent to utilize available staffing
resources, the City must also strategically invest in external resources for a successful roll-out.
On June 19, 2023, the City Council approved the FY 2024 CIP Budget with the new FTTP project,
and Grid Modernization for Electrification Project. The approval of the electrification project
accelerated efforts to align electrification and fiber construction, which impacted the Fiber
Expansion Plan. Staff is deploying a pilot to determine how to align the grid modernization project
and projects under the Fiber Expansion Plan to help minimize utility engineering pole make-ready
work, pole replacements, noise disruption, and construction activity in neighborhoods.
2 https://cityofpaloalto.primegov.com/meeting/document/1960.pdf?name=Item%208%20Staff%20Report
ANALYSIS
Align fiber and grid modernization projects
The City conducted a high-level electrification study to evaluate the impacts of projected
electrification loads on Palo Alto’s distribution and substation transformers, primary/secondary
distribution circuits, and to propose upgrades needed to mitigate overloads. The estimated cost
to construct the necessary electric system upgrades for a 100% electrification scenario is
between $220 million to $306 million, depending on the approach. The workplan for this project
is a staged, multi-year approached designed to accomplish the upgrade of the electric
distribution system to meet the City’s goal of being ready for full-scale electrification by 2030.
Staff is in the process of implementing a pilot project to convert a 4kV area to 12 kV and upgrade
the transformers and secondary networks to facilitate electrification. This work is expected to be
completed by the middle of 2024, depending on other factors which impact project timelines,
such as material lead times, pole make ready, and alignment with the fiber projects. The initial
purpose of the pilot was to determine the feasibility of electrification design and construction
methods and implement best practices to facilitate the most cost-effective deployment of
resources. This purpose has since been expanded to also test aligning the Fiber Expansion Plan
with electrification efforts. Staff will be analyzing the cost savings, timeline, and resources
(staffing and contractors) required for project alignment.
Given the hundreds of miles of overhead and underground construction activity between FTTP
and electric grid modernization, the pilot enables staff to explore various construction phasing
options to minimize disruption, construction activity, and construction costs within
neighborhoods while avoiding prolonged deployments. In addition, aligning these projects in the
pilot helps alleviate construction constraints as staff does not have enough internal resources to
project manage, perform engineering make ready, and inspect construction for both projects in
parallel.
California Environmental Quality Act (CEQA) Compliance
The City will conduct an Initial Study on the environmental impacts associated with the
construction and operation of the potential citywide FTTP network, will cross much of the City of
Palo Alto and include some new infrastructure (i.e. fiber hut, fiber cabinets, aerial and
underground cables, utility vaults). The Initial Study is estimated to take about 30 weeks to
complete, including internal and mandated review time such as 30 days for public review and
comment.
The preliminary assessment, or Initial Study, will evaluate the project’s potential impacts on
various environmental factors and identify whether those impacts can be mitigated. If the
project’s impacts can be mitigated to a less-than-significant level, then the City may issue a
Mitigated Negative Declaration (MND) or Negative Declaration (ND). If the preliminary
assessment of the project determines one or more significant impacts on the environment
despite mitigation, a full environmental impact report (EIR) will be required, involving a more
comprehensive and detailed analysis of the project’s impacts and alternatives.
The City has retained a CEQA consultant to prepare a CEQA Initial Study in accordance with local,
state, and federal statutes. The Initial Study identifies physical, biological, social and economic
factors that might be affected by the fiber expansion project. Some environmental factors that
could potentially be affected by this project include aesthetics, air quality, geology/soils,
hazardous materials, noise, utilities services, and hydrology/water quality. Specific examples of
likely CEQA analysis include:
o Hydrological analysis to see if any proposed fiber lines would run through areas
near bodies of water and show that there would not be significant impacts to
waters of the state or waters of the US, as well as provide recommendations for
how to protect nearby waterbodies even if there are no direct impacts (e.g.
through proper stormwater protection measures)
o Air quality analysis (including understanding how to run claimed calcs for the
proposed project to evaluate air quality emissions and GHG emissions)
o Noise modeling (including analysis of potential impacts on nearby sensitive
receptors from construction noise and vibrations)
o Hazardous materials analysis (including evaluation and mitigation if necessary, of
potential impacts to construction workers and sensitive receptors especially if any
of the dark fiber runs through areas of the city runs through a plume)
Replacement of Jointly-Owned Poles
The City jointly owns 5,550 poles with AT&T. The current pole intent process (including billing)
for managing joint pole replacements operates under a joint pole agreement executed between
the City and AT&T in 1918. This process may not be feasible to accommodate the increased
volume of pole transactions expected with the grid modernization and Fiber Expansion Plan.
CPAU began discussing with AT&T effective ways to coordinate and collaborate on the
anticipated surge in pole replacement work and mitigating the impact to the community. One
potential improvement is to identify third party resources authorized to work on both AT&T and
CPAU facilities during pole replacement, make ready work, and/or inspection.
As part of the effort to streamline processes, CPAU also reevaluated the benefits of joining the
Northern California Joint Pole Association (NCJPA). Members have “joint pole equity” and
standardized cost-sharing methodologies for pole ownership, maintenance, use, setting,
replacement, dismantling, relinquishment or removal of jointly owned poles. CPAU has operated
without being an NCJPA member as the volume of pole replacements were manageable under
the existing process. The main objectives for considering membership with the NCJPA were to a)
streamline the pole intent and billing process; and b) improve the process of recovering AT&T’s
costs for joint poles replaced by CPAU. After further evaluation, staff concluded neither
objectives would be met by joining NCJPA as lead times are longer under NCJPA’s pole intent
process, and there are more administrative steps for limited cost recovery. For example, the
current joint agreement with AT&T requires a response within 10 days vs. 45 days according to
the NCJPA handbook, and the current agreement is based on actual replacement costs while
NCPJA joint pole costs are based on average unit costs for all members.
Design and Construction Contracts for Pilot Area
Given the anticipated increase in pole work, CPAU will need to add resources to manage and
perform a high volume of pole make-ready work and replacements. In the meantime, staff is
evaluating how to leverage existing on-call construction and engineering contracts to perform
the engineering design, overhead and underground construction, and installation for the pilot
area (approximately 409 poles and 1,224 homes). This will reduce the construction timeline by 6
– 9 months, which would otherwise be prolonged in the process of issuing multiple invitation for
bids (IFBs) to perform the work for FTTP and grid modernization. The pilot area will be treated as
a testbed to design and construct both projects in parallel to minimize community disruption and
reduce shared construction costs. The pilot will inform CPAU how to most effectively bid out the
engineering, construction and installation services required for FTTP and grid modernization for
the remaining phase 1 area (additional 1,241 poles and 5,560 customers).
•VIP Powerline provides electric construction services including pole replacements,
facilities upgrade on the electric distribution system, and customer connections. VIP also
has some fiber construction experience such as installation of fiber cable and messenger
wires. (Staff Report 2303-1119)3
•MP Nexlevel provides electric and fiber substructure installation and trenching services.
MP Nexlevel is also qualified to provide fiber construction and installation services
including fiber installation, splicing, and testing. (Staff Report #13953)4
•Magellan provides multiple facets of implementing municipal fiber and broadband
networks including business planning, engineering, program management, integration,
and operations. In 2022, Magellan was acquired by Entrust Solutions, an experienced
provider of engineering, consulting, data analytics, and automation services to the
electric, gas, and telecom markets. (Staff Report #2303-1215)5
After staff identifies the specific build of materials (BOM) for electric and fiber in the pilot area,
staff will submit the BOM to the existing vendors to receive quotes. Staff will evaluate the bids
and return to Council with contract amendments for the additional scope of work and updated
not-to-exceed dollar amounts.
Other Updates
Fiber Make-Ready Engineering
Magellan completed the survey of approximately 6,000 utility poles in Palo Alto. The survey
identified the current load and condition of each pole in preparation for the new fiber backbone
and FTTP. The survey included detailed fielding and walk-out of all routes to validate running
3 https://cityofpaloalto.primegov.com/meeting/document/1862.pdf?name=Item%2010%20Staff%20Report
4
https://cityofpaloalto.primegov.com/Public/CompiledDocument?meetingTemplateId=3685&compileOutputType=
1
5 https://cityofpaloalto.primegov.com/meeting/document/1960.pdf?name=Item%208%20Staff%20Report
lines, existing utilities, and constructability. For aerial construction, Magellan provided make-
ready engineering pole data to determine pole preparation and pole replacement, as well as
timeframes for the work to be completed. For phase one of FTTP and grid modernization, CPAU
will be passing 1,650 existing poles, attaching fiber to 1,300 poles, replacing 200 to 300 poles,
and coordinating with third parties to move their telecom equipment lower on 325 poles. In
addition to the fiber expansion project, CPAU has a recurring Wood Pole Replacement capital
improvement project to replace approximately 100 deteriorated wood poles annually. Wood
poles are used to support overhead utility lines throughout the City of Palo Alto. Though poles
are inspected, tested, and treated to maintain their integrity, over time poles will deteriorate to
the point that they are no longer in compliance with GO 95 requirements or safe for community
and utility workers.
Fiber Hut Sites:
Fiber huts are usually a prefabricated building (10’ x 20’) containing electronics and network
equipment that connects fiber to neighborhoods. The hut also has backup generators and HVAC
systems to maintain equipment within its operating environmental specifications. Two fiber huts
are recommended for the citywide network for Palo Alto Fiber. Staff and Magellan evaluated
potential fiber hut sites and considered their proximity to strategic areas, planning/land use
requirements, and existing infrastructure. As a result, staff narrowed down the potential
locations to the Colorado substation and anticipated space at City Hall. The Colorado substation
was previously identified as an ideal location due to its proximity to the area it serves and its
current use for utilities. City Hall is anticipated to have vacated space after the Police Department
moves to the new Public Safety Building. City Hall would be an ideal location for a secondary hut.
Due to the centralized location and potential square footage, this site may also be considered for
a future data center, further securing the future of the City‘s data infrastructure.
Equinix Fiber Data Center:
Equinix is currently the hub for the City’s dark fiber service. The Equinix datacenter is a vendor-
neutral site where numerous internet service providers and content delivery networks can
interconnect and exchange internet traffic between their networks. As the City expands its fiber
footprint, more data center space will be needed. For FTTP, the City will have to increase the
number of leasing cabinets (vertical racks sited next to each other, each capable of holding
multiple termination/switching boxes in stacked configurations) from five to 12 cabinets. The City
is undergoing discussions with Equinix to lease additional space available in their Palo Alto
location and to explore other Equinix locations. The City may also consider increasing the size of
the data center at City Hall.
Invitation for Bid (IFB) Construction Package:
When opportunities arise to align FTTP and grid modernization, CPAU and Magellan may gather
and consolidate all supporting documents to bid out construction of FTTP and grid
modernization. The consolidated IFB package will include construction-ready prints for fiber and
power, construction details, splice details, pole make-ready details, construction standards as
well as all documentation needed to complete the bid package. This shall include a written
summary and scope of work, estimated costs to construct, as well as labor estimates, bill of
materials, and the vendor list.
Staffing:
As the City solidifies staffing plans in parallel with contracted services, the Director of Information
Technology will take on the responsibilities of an Assistant Director for Palo Alto Fiber on an
interim basis. In the FY 2024 adopted budget, four (4) new FTE positions for the dark fiber
expansion and implementation of FTTP were created. These positions will be recruited and filled
as needed during the various stages of the project.
•Assistant Director - To provide the vision and strategy for the new fiber business and lead
a high-performance team to quickly execute. Responsibilities include and are not limited
to: overseeing fiber enterprise fund budget timelines and milestones (including the dark
fiber optic business), managing the roll out and expansion of the fiber optic network,
Internet Service Provider (ISP) operations, customer service operations, and business
development.
•Manager Utilities Telecommunications / "Outside Plant Manager" - To lead the
construction process, installation and repair service technicians, and ensure the outside
plant processes run efficiently and smoothly. Responsibilities include and are not limited
to: overseeing construction, managing engineers and contractors, achieving construction
budget timelines and milestones.
•Manager Information Technology / "Sr. Network Engineer/Architect" – To lead the
development process of the overall architecture of the broadband system and manage
the system. Responsibilities include and are not limited to: managing the network and
supporting technical needs across the organization.
•Manager Utilities Telecommunications / "Sales and Marketing Manager" - To develop
the marketing strategy and lead marketing initiatives. Responsibilities include and are
not limited to: meeting revenue objectives, cultivating strong partnerships, and
maintaining retention rates.
FISCAL/RESOURCE IMPACT
This report is for informational purposes so there is no resource impact. Based on Council and
UAC input, staff will return with specific actions associated with efforts to align the fiber
expansion and electric grid modernization projects.
STAKEHOLDER ENGAGEMENT
On November 2, 2022, the UAC unanimously recommended to build fiber backbone and FTTP
under a phased approach with existing funds ($34 million from Fiber and $13 million from
Electric). The UAC expressed the goal of FTTP is to provide ubiquitous or citywide high-speed
internet access to all residents and businesses in Palo Alto. If financially self-sustaining, and
deemed successful, the first phase of FTTP can become a springboard to a citywide FTTP
deployment within five years.
On December 19, 2022, City Council directed staff to proceed with the Fiber Expansion Plan to
implement the Fiber Rebuild project and Phase 1 of the Fiber-to-the-Premises (FTTP) project
without debt financing. Included in the Council’s motion was direction to: a) maximize number
of homes and businesses passed; b) consider promotional rates to increase take rate; c) define
leading indicators and metrics to determine success; and d) recommend future Council action to
accelerate expansion if metrics are positive, including a potential bond to streamline construction
and compress construction time as much as feasible.
ENVIRONMENTAL REVIEW
The Finance Committee’s discussion of the Fiber Expansion Plan project and its potential
alignment with the grid electrification project is not a project requiring California Environmental
Quality Act review, because it is an administrative governmental activity which will not cause a
direct or indirect physical change in the environment. CEQA Guidelines Section 15378(b)(5).
ATTACHMENTS
Attachment A: Presentation
APPROVED BY:
Dean Batchelor, Director of Utilities
Staff: Dave Yuan, Strategic Business Manager
Palo Alto Fiber Project and
Construction Alignment
with Electric Grid
Modernization
November 7, 2023
-2
AGENDA
•Project(s) Recap
•Fiber Expansion Plan
•Fiber Backbone Expansion “Rebuild” Project
•Fiber-to-the-Premises (FTTP) Project
•Electrification
•Grid Modernization for Electrification Project
•Informational Updates
•Alignment of Fiber Expansion Plan and Grid Modernization
•California Environmental Quality Act (CEQA) Initial Study
•Joint Pole Ownership Coordination with AT&T
•Contract Amendments
•Other Updates
-3
HOW DID WE GET HERE?
•What has Council Approved?
•Dec 19, 2022 Council Approved to
•Proceed with dark fiber backbone
expansion “rebuild”
•Build FTTP Phase 01 in multi-
phased approach without debt
financing
•Jun 19,2023 Council Approved
•New Grid Modernization for Electrification project in FY24 Budget
•New Fiber-to-the-Premises project in FY24 Budget
-4
RECAP OF PROJECTS
FIBER BACKBONE REBUILD FIBER-TO-THE-PREMISES GRID MODERNIZATION
Estimated cost $25.6M (FO-16000)Estimated cost for Phase 1 $20M;
Citywide $102M (FO-24000)
Estimated cost $220M to $306M
(EL-24000)
Funded by Electric and Fiber funds Funded by Fiber Fund Funded by the Electric Fund; debt
financing and/or grants
Supports the City’s fiber licensing
service and internal needs:
Improve fiber and electric
system reliability, security
and capacity
Enable more connectivity for
City departments
Internal communication
needs for SCADA, AMI, DA
and other critical
infrastructure
Supports the City’s high speed
broadband internet needs:
Provides City-owned
high-speed internet
service
Provides new capacity for
dark fiber licensing
Community investment in
its own broadband
network
Supports City’s S/CAP goals and
future demand
100% residential
electrification
Upgrade existing
substations,
replace transformers,
convert existing 4 kVa lines
to 12 kVa to increase
capacity and meet
future demand.
-5
INFORMATIONAL UPDATES
Topics to address prior to construction
Pilot aligning fiber and grid modernization project to inform on
•Pole make-ready and replacement
•Material lead time and warehousing
•Internal resources to implement and coordinate projects
Initial Study for CEQA in progress to evaluate potential impacts
•Identify physical, biological, social and economic factors
•Provide mitigation efforts
Joint Pole Ownership Coordination with AT&T in progress
•Approximately 5,550 poles shared with AT&T
•Upcoming replacements and coordination will overwhelm staff
•Eliminated Northern California Joint Pole Association membership option
Contract Amendments
•VIP Powerline –power and fiber construction
•MP Nexlevel–substructure and fiber construction
•Magellan/Entrust–power and fiber engineering design
-6
Pilot Map Overlay with Phase 1 & Areas of Interest
Pilot Area Map to align Grid Modernization with FTTP
(1,224 customers and 409 poles)
Proposed Phase 1 FTTP Map (Additional 5,560
customers and 1,241 poles)
Areas of Interest Map based on Lower Construction
Cost, Higher Density, Less Competition, Higher
Customer Interest
- 7
Pilot Project Plan (draft outline subject to change)
Pre-construction
•Aerial Power Make Ready Engineering - Feb 2024
•Materials Ordering/Warehousing - Feb 2024
•CEQA Approval - Apr 2024
•Fiber Hut at Colorado Power Station - Oct 2024
Construction
•Power: Mar 2024 - Oct 2024
•Fiber: Jun 2024 - Feb 2025
Operations & Customer Outreach
•Operations Support Systems “OSS”
•Business Support Systems “BSS”
•Marketing/Event(s)
•Community Meeting(s)
-8
Q&A