The SIT Subcommittee identified five priority areas from this list, for further consideration as system indicators. The priority areas are:

a. Market Penetration

b. Return on Investment (ROI)

c. Total System Investment

d. Customer Satisfaction

e. Customer Repeat Usage

The following presents the SIT’s recommendations regarding each of these areas along with the underlying background discussion

Market Penetration

Recommendation

The ability to measure the market penetration rate for both business and jobseeker customers is an important basic goal of the system. Furthermore, the development of market penetration measures would require partners to begin sharing customer information in a common database, which in turn would promote integration.

Market Penetration Rate – Business Customer = Business Customers

Total Number of Businesses

Market Penetration Rate – Jobseeker Customer = Jobseeker Customers

Local Civilian Labor Force

Discussion Points

Tompkins -- Business Customer base (business who has accessed our services/activities in the last 3 years) and total number of firms (business with as least one employee and pays into UI).
Sullivan -- Said this would be 10 employees or more and active employers (job fairs)
We can start to look at diversity or what it is we want to count in the universe. What data is available? We don't have VESID or other partner numbers: Employers need to be put in the database-and post their business contacts.
Total number of firms in that area
What firms do not want to participate in the public employment system?
Should we/can we count in the denominator those firms that are hiring or doing job development work, and then look at how many we served.
Which jobseeker customers would be counted in the denominator? Does this include all customers who have received some sort of employment related service by any of the partner agencies? For example, would it include TANF exhaustees who have been hired to work temporarily for the NYC Parks Department or participants of a Food Stamp Employment and Training program?
Is there the potential for localities to be selective in determining which jobseekers would be included so that they could maximize or minimize the rate? Which raises the question, “which is better, a high or a low Total System Investment Rate?” Could a high rate be better in some systems and not as good in others? Should the total system investment be linked to a specific outcome such as employment?
New hire data is available. Is this a subset of the first?
Are we utilizing resources correctly? When we look at hiring do we exclude those that just use the union halls?
In terms of LMAs or information what can we collect? DOL’s (DoES) definition of what can be collected is: personal visit; telephone conversation; job listing; job fairs?
We need to ensure that there is no duplication of the customer by multiple partners. Active businesses over what period? What can we reasonably access? The measures should be encouraging the use of a system wide collection of business services etc. Business wants a single point of contact. Therefore, we conclude that there needs to be a single data system. We need to follow the principles that we have established and then work towards the end goal of partners’ ability to report on these indicators. We need to ensure that all services: Core service, intensive and training services are done with an account manager.

A subset for Market Penetration: # of Firms (numerator)

# of Firms Hiring (denominator)

Problem - How to collect numerator data from areas with multiple Business/Employer databases? Is this a data collection issue or a definition issue? Integrated outcomes are the key and the desired goal. How do we define what an active business is? Time and services/activities: what constitutes the numerator?
First find out what businesses want to know about the system. The concept of market penetration is interesting. However presenting such a measure as a simple fraction with huge differences in the numerators and denominators is neither useful nor useable. A measure such as this would require the development of a regression formula that accounts for the weight of numerous variables within the labor force. Will this be useful to system users?
Some business partners have indicated that we should focus on: how many people were served and how much did it cost?
Services to business: define what you are going to count and not count. Some services are being provided but we are not getting credit.
Issue - Do we have the number of 14 and 15 year olds that have working papers? Do we include 14 & 15 year old

Return on Investment (ROI)

Recommendation

· Do not develop a system indicator for Return on Investment (ROI), as the methodology and underlying assumptions for computing ROI are highly subjective.

· Instead, focus on examining Total System Investment across the business and jobseeker customers.

Discussion Points

· Calculating Return on Investment (ROI) of a public program requires placing a monetary value on both the cost of the program and the benefits derived from the program. The methodology for computing these values is highly subjective, and gives rise to results that can vary considerably based on changes in the underlying assumptions about the costs and benefits attributable to system programs (e.g., reduced crime, reduced dependence on Family Assistance, Medicaid, Food Stamps, reduced UI duration, State tax revenues, economic multiplier effects attributable to employment income, etc.).

· Most ROI studies tend to overstate the benefits relative to the costs.

· On return on investment measure... it appears to focus on cost per customer... my concern is the natural tendency for people to look at the lowest cost as the best, which won't necessarily be the case for our work. A lot of cost may be spent for a return of a significantly higher skilled and paying job.

· The feds are moving in this direction, so the ROI measure would be good to watch... but maybe it could be balanced with one that includes the value of the outcome, be it wage or skill level.

· Business: What is the business investment? Is it investment of their time? Is this is a return of our investment or of the business? Can you really measure what resources we invest in business? What is our return? What are we looking for as a return/benefit? Suggestions: Tax savings; UI savings; savings of criminal justice costs; return to the economy (multiple benefit to the local economy); costs to business in providing internal human resource activities; lower the UI rating?

· Community Investment (tax dollars invested) and community benefit (tax dollars saved or new tax revenues). Is ROI the same for businesses and jobseekers? Is it ROI or value added to the business? Benefit doesn't always accrue to the business that used it.

· What is the Core service level that every partner must provide? Cooperation; collaboration and integration? Referrals to and from the Center? What agencies are entering businesses and jobseekers activity into the common database? How many referrals does each partner make?

· Suggestion - Scrap the ROI indicator and instead measure System Integration. Value added/Benefits received? Community Colleges generate funds for enrolling individuals –EPE and FTE. We need to communicate that the benefit that the CC receives by generating these funds are as a result of the referrals/participation within the system. Therefore, there are sometimes benefits that we can identify that the partners might not see as a benefit received from their participation in the system. (R&S will look into this).

· Conclusion - Statement about why we do not think that ROI is a reasonable system indicator: We should start with System Integration and benefits received from this integration. What do we look for in system integration that could be measured: common intake for all customers; testing and assessment; job development/ placement; one point of contact?

· Businesses want to see a business broker. Integration allows a single point of contact that packages the partner services. How do we equate or value this?

· The Team believes that if you do well on these things we are improving the ROI for system participation. How do you measure the success? How long did it take to serve the customer? As we get better at integration- did this take less time; what was the cost of OJT; human resource product (what was the saving to the business?). Integration could mean less referrals but better customer satisfaction and intensive services.

· Need to be able to obtain Total System Investment. How do you measure Total System Investment?

Total System Investment

Recommendation

The ability to measure and assess Total System Investment across business and jobseeker customers is an important basic indicator of the system. Furthermore, the measurement of Total System Investment measures would require partners to begin sharing information, which in turn would promote integration.

Total System Investment Rate – Business Customer = Total System Investment

Business Customers

Total System Investment Rate – Jobseeker Customer = Total System Investment

Jobseeker Customers

Discussion Points

· Things to think about: how do you calculate system investment/costs? Adding value added? Possibly using the mandated partners only?

· Total System Investment across Business and Jobseeker customers should not be interpreted as a measure of efficiency.

Customer Satisfaction

Recommendation

· Do not develop a specific customer satisfaction system indicator for business customers and jobseeker customers. Instead, use system indicators such as repeat usage, market penetration and referrals from customers as indicators of customer satisfaction.

· Customer Satisfaction is most meaningfully measured through immediate customer feedback, which is outside the scope of a system indicator.

Discussion Points

Business Customer Satisfaction

· If we can decide on how to survey businesses to see how satisfied they are with the services/activities with in the system, would DOL be able to handle?

· Do not use surveys to indicate business satisfaction. Indicate business satisfaction by the number of businesses that are using repeat services; or referred others to the system?

· Do all local areas want the State to control this process? State administrated and funded?

· Since the state is already funding the 3 Customer Satisfaction Survey Questions mandated in WIA could the local area add their own questions to that same process?

· Should it be done twice a year?

· Confidentiality is an issue for both the jobseeker and business side.

· Could this be a Web-based application?

· Surveys may not be the best way to go. Do customers really tell us what we need to know for continuous improvement in a survey?

· What other means can be utilized to collect/receive customer satisfaction?

· The surveys that are returned are usually only those that are willing to respond-those might be only those that have positive feedback (afraid to contribute negative comments). How valuable for continuous improvement would this information be?

· How are the surveys distributed?

· System-wide/state-wide, how do we do this meaningfully?

· On the business side, what is a happy customer?

· Important information: how did you hear about us? (Marketing efforts)

Jobseeker Customer Satisfaction:

· Focus on the system not the Center

· It is difficult to obtain meaningful jobseeker customer satisfaction information.

· Obtaining immediate customer feedback is more meaningful and useful for continuous improvement.

· Indicators to see if quality increased: use of the website; use of center; number of partners encouraging the use of the system as quality service.

· Market penetration is a proxy measure of customer satisfaction.

· The only other method or indicator that might be useful would be completion rate. But since completion is not the goal of all of the services provided, this would not be the best count or outcome.

· It is difficult to obtain a customer satisfaction survey and process that will draw meaningful conclusions.

· Do not develop a system indicator for Customer Satisfaction for the jobseeker. Obtaining immediate customer feedback is more meaningful and useful for continuous improvement. Similarly, market penetration is also a proxy for jobseeker customer satisfaction.

· Since most of the data that is collected is irrelevant, we should stay to the market penetration.

Business Customer Repeat Usage

Recommendation

The ability to measure repeat usage for the business customers is an important indicator of customer satisfaction. Furthermore, the measurement of business customer repeat usage would require partners to begin sharing business customer information in a common database, which in turn would promote integration.

Repeat Usage Rate – Business Customer = Repeat Business Customers

Business Customers

Discussion Points

· Refer to discussion points under Customer Satisfaction.

Establishing Operating Principles

Numerous operating principles for developing system indicators were identified through SIT and Local Workforce Investment Board (LWIB). These principles fell into the following nine areas:

1. Development versus Performance

2. System Focus

3. Customers

4. Data Collection

5. Report Cards

6. Continuous Improvement

7. System Integration

8. Focus on Success

9. Sharing Outcomes (Successes)

The following presents each operating principle along with background discussion points.

Development versus Performance

Operating Principle

The indicators should focus on system development rather than system performance.
The indicators are a starting point for working towards (developing) more refined indicators of system performance.

Discussion Points

There is a subtle difference between “performance” indicators and “development” indicators.
The ability to effectively measure system performance and to establish expectations based on this performance requires the collection of system data that is uniform, verifiable, consistent, readily available, conclusive and easily understood across the system. The current state of system data is not sufficient to effectively measure and gauge performance.
Development indicators allow for assessment of how well the system is developing, and provide a starting point for working towards creation of performance indicators.
Development indicators would provide a starting point for creating more refined indicators that can be used for purposes of measuring and gauging performance.
This could be accomplished once we have everyone collecting/integrating. Refinement of data will come later as information is collected on a system-wide basis.
We need to point to the principle that the indicators will encourage continuous improvement.

System Focus

Operating Principles

The indicators should have a Statewide focus, and reinforce “system”, not “center” or “program.”

The indicators must be relevant to the State Workforce Investment Board (SWIB), Local Workforce Investment Boards (LWIBs); Chief Local Elected Officials (CLEO) and system partners.

Discussion Points

Statewide reporting focus to reinforce system, not center
Need to define what is relevant across the state and communities despite differences in local economies.
Need for information that is useful at all levels.
The information can be aggregated at the state level.
Need to be aware of differences between system indicators and programmatic outcomes, and focus on system indicators.
Focus on State, and then there can be unique indicators at the local level that will take into consideration local factors.
WIBs prefer macro level, with good cycle time and not tied up in process.
Do you mean Statewide or local system? Systemwide indicators may not reveal where underperformance exists and limit their usefulness in targeting remedies.
If the focus is Statewide, relevancy to individual LWIBs and CLEOs will be limited.
WIBs and community want to know impacts of the system that indicate improvement: a lower unemployment rate; impact on welfare; size and growth of labor force; economic development.

Customers

Operating Principle

The two primary customers of the system are:

o Business; and

o Job Seekers (e.g., emerging, transitional and incumbent workers).

Discussion Points

Business should be considered the primary customer, if employment is viewed as the primary outcome of the system (e.g., if business is not “buying” system services, then there is no system). In this case, a qualified workforce (i.e., job seekers) is the primary product that the system is providing to the business customer.
Some system partners provide services to individuals, where employment is not the primary goal.
Jobseeker customers are those that receive core, intensive and training services and have been served by any of the mandated partners.

Data Collection

Operating Principle

There must be a strong commitment by all Local Workforce Investment Areas to put the core data elements into a common database (electronically or otherwise), in order to establish consistency in measuring and interpreting indicators.
The data collection process to support calculation of the indicators must not be onerous on system partners.

Discussion Points

What do we mean by the indicator?
Is anyone measuring it now, and if so, how?
Must be able to un-duplicate information
Common data elements
Obtaining unduplicated jobseeker customer counts will be a challenge if all the partners do not use the same database.
LWIBs want to know what data is already available at the state level in order to access it.

o If data is available already, then not a problem, but LWIBs do not want more requirements/mandates.

Some partners are going to be reluctant to cooperate/participate and provide the underlying data necessary to compute the system indicators (i.e., they will not want to count anything beyond what they are already counting).
Will all partners in a LWIA provide the data to the LWIB and the LWIB forward the data in a prescribed format to the common database? Is this feasible for dollars spent by each partner? Are data elements defined consistently from program to program or are there differences that impinge on the usefulness of consolidating data from several partner programs?
The onerous part may be in consolidating data from numerous data collection systems.
We need to develop what constitutes the numerator and denominator for each of the indicators. Then we need to explore how we get the necessary data?
The underlying data used to calculate the indicators should be in a single database.

o OSOS database or some other that all partners can utilize.

Integration of information/outcomes (non duplication) into a single database is more important than the specific data collection system being utilized by the local area.
Do not attempt to create a single data system to be used by businesses and partners. A "one size fits all" approach will not work (for example, OSOS) and will divert valuable resources away from the task without fostering system integration.
Find out what data is already being collected. Do not limit yourself to what NYS is doing. The Feds collect and analyze volumes of data on employers that might be useful. Other sources such as local utilities and regional business organizations might be good sources of data.
If additional business or partner data needs to be collected let the local Board/Partnership do it and send to a central point for analysis.
Consider the use of sampling.
FMS WDB has developed a software package and database (starting with 600 local employers with 10 or more employees) to allow all members of our business services consortium to update or share new information on outreach business including business needs, employment t opportunities, and ways to market all programs and funding opportunities. access is available to consortium members through the FMS web site.

Report Cards

Operating Principle

The indicators should lend themselves to a system report card (e.g., NYSED report cards).
System report cards should provide for local context for the indicators (e.g., local economic conditions, etc.).

Discussion Points

Exec. Committee of Board: Partner agencies are reluctant to report, and fear that if we ask for some additional information they will not have the staff or resources to do it.
The report card is not intended to compare indicators across local areas. It should only report the indicators for the local area.
Provide local context for the indicators (e.g., local economic indicators, etc.).
As currently defined, do not control for exogenous variables such as economic cycles. The number of customers in need of service may vary significantly with swings in the economic cycle. The number of customers should be measured against the number in need of service, not the universe of businesses or the labor force. These are indicators not measures. They could change each year.
They need to include general local economic factors (e.g., unemployment; jobs gained or lost; welfare caseload; poverty; wage rates; etc.).
Expectations about report cards must be aligned across system partners.
LWIBs must identify statistics (indicators) that are important to them.
Some partners want to know the benefits; time, money and return.
This equates to numbers served but how do we capture all that use the system outside of registration/staff assisted?
Develop and submit a set of templates for a partner program report and system success measures and a system report.
Products developed should be multi-tiered:

o LWIB and CLEO: internal

o Community within local area-external

o SWIB and State Elected Officials-comparative by 11/15/02

What is a report card and what should be included in a system report card (from LWIB Retreat on July 16 & 17):

o Report could be individual partner measures/outcomes and allow for data that the private sector is looking to capture.

o It is important to look at the impact of the system goals vs. individual partner goals.

o Whose measures are we using? Federal, employer? In relation to what?

o Who is using the information? If we want business to use and value the system, then we need to capture information that highlights our successes.

o We need to capture core services.

o Boards want to know about the value that services are bringing to the community.

o Performance of the individual partners can be common ground. We need to learn what it is they measure and ensure that we are not making this so complex.

o Ask the private sector what they want in performance outcomes.

o Localized information can be added if the local board has additional needs.

o We should ask the partners for a "wish list" of what they would like to see as system outcomes/report. This would help get them to buy-in to the system report. They should indicate what benefits this would have to marketing their program services.

§ Summary:

· It was agreed that we need to track the return on investment method;

· that a report card would reflect all of the partners outcomes;

· areas can add other elements that would meet local needs;

· We need to decide if we would limit the system report card to only the partner performance outcomes that are the same or similar across all partner programs.

· How do you count incumbent worker outcomes? What partners contributed? What partners have the same outcomes?

· Who are the audience and the use?

· Is there different data/or level of data available for different purposes and groups?

· If there are any other initiatives that are looking at the system performance, we need to coordinate and keep each other informed.

· We will develop a system report card for three levels of customers: LWIB/CEO (internal) ; Community (external) ; SWIB/state/federal (comparative).

· Measure outcomes, not just output. Must look at quantitative and qualitative.

Continuous Improvement

Operating Principle

The indicators should promote continuous improvement.

Discussion Points

The purpose of the indicators is continuous improvement and return on investment.
There will be the tendency to evaluate whether the outcome is “good” or “bad”.
There will also be the tendency to rank/compare local areas based on the indicators.
It must be stressed by the SWIB that determining “winners” and “losers” is not the purpose of development indicators. Rather, it is about measuring and making improvement based on the data and local area priorities.
If comparisons are to be made, they should only be made across substantially similar areas to see what can be done to improve their outcomes.

System Integration

Operating Principle

The indicators should promote system integration.

Discussion Points

Discussion from July 16 & 17 SIT meeting

Focus on Success

Operating Principle

The indicators should be designed to highlight success rather than failure.

Discussion Points

Develop a template for system success measurements.
If the indicators are to be taken seriously, they should detect both success and failure. Of course, if the purpose of the indicators is to develop positive publicity, then it makes sense that only successes should be measured.

Sharing Outcomes (Successes)

Operating Principle

All system partners would share system outcomes (successes).

Discussion Points

Partners are concerned that their performance gets blended with the other partners.

o OSOS is a problem - if all of the partners were using this next step would be easier. They don't have a lot of faith in the technical aspect of the system.

Attachments

Attachment 1 - LWIB Retreat, July 16-17, 2002

In attendance: Deborah Giordano (Tompkins); Alice Savino (HMO); Laura Quigley (Sullivan); Joan Sinclair (Cat/Alley); Jim Calnon (North Country); Judy Davison (Cay/Cort); Gail Breen (FMS); Karen Springmeier (Finger Lakes); Kevin Price (CDO); Abe Sabbas (Hempstead); Butch Rehm (SWW).

Note Taker: Karen Coleman (NYSDOL)

Observers: Greg Newton; Russ Simon

Discussion Notes:

Group commented on what is a report card and what should be included in a system report card:

Report could be individual partner measures/outcomes and allow for data that the private sector is looking to capture.
It is important to look at the impact of the system goals vs. individual partner goals.
Whose measures are we using? Federal, employer? In relation to what?
Who is using the information? If we want business to use and value the system than we need to capture information that highlight our successes.
We need to capture core services.
A board wants to know about the value that services are bringing to the community.
Performance of the individual partners can be common ground. We need to learn what it is they measure and ensure that we are not making this so complex.
Ask the private sector what it is they want in performance outcomes.
Localized information can be added if the local board has additional needs.
We should ask the partners for a "wish list" of what they would like to see as system outcomes/report. This would help get them to buy-in to the system report. They should indicate what benefits this would have to marketing their program services.

In summary:

It was agreed that we need to track the return on investment method;
that a report card would reflect all of the partners outcomes;
areas can add other elements that would need local needs;
We need to decide if we would limit the system report card to only the partner performance outcomes that are the same or similar across all partner programs.
How do you count incumbent worker outcomes? What partners contributed? What partners have the same outcomes?
Who are the audience and the use?
Is there different data/or level of data available for different purposes and groups?
If there are any other initiatives that are looking at the system performance, we need to coordinate and keep each other informed.
We will develop a system report card for three levels of customers: LWIB/CEO (internal) ; Community (external) ; SWIB/state/federal (comparative)
Measure outcomes, not just output. Must look at quantitative vs. qualitative.

Discussion on some of the areas to measure for the 2 primary customers (Employers and Jobseekers):

Employers:	Jobseekers:
Market Penetration:	Entered employment
Number of people trained Kind of jobs that are filled/trained for Cycle time Usage/Growth Rate (last year vs. this year)	Both
Market Diversity	Both
Customer Satisfaction	Both
ROI	Both
Referral rate	Both Decrease in public subsidies Recidivism

Job Order Fill rate? or are we interested in the number of job openings in the local area and are ability to fill those jobs? Job order fill rate vs. job opening fill rate.

Day 2
Consensus on the following:

New Name: System Indicators Team (SIT)

Broad Principles

Develop and submit a set of templates for a partner program report and system success measures and a system report.
Products developed will be multi-tiered:

LWIB and CLEO: internal
Community within local area-external
SWIB and State Elected Officials-comparative by 11/15/02

The purpose of the products will be continuous improvement and return on investment.

Process: State Level Focus

Involve:

Missing Team Members

Partners (local)
Partners (state-level)
State-Level Partners Table (System costs group)
OSOS workgroup
LWIB - what they would like to see
An R&S expert
Other LWIB Directors

Timeline: See attachment

November 15, 2002 - Due to WIA Reauthorization Committee

Assignments

Staff support: Karen C and R&S
Subcommittee: Judy; Alice; Joan; Deb; Laura; Kevin; Hempstead : Ask for other members
OSOS Link: Gail and Karen S.
State Group Liaison: Karen C.- Ask for state reports from all state partners
State Web Board
All WIB Directors will collect info. From Local Partner and WIB input members

Packaging

SDOL and Patrick Doyle will help to promote a marketing package.

Homework:

What Information/what are we going to ask from our partners:

State Wide Reporting Focus
What indicators promote continuous improvement and return on investment
What indicators would add value to your organization/customers
Employer and Jobseeker Focus
Reinforce System, not center
Success Indicators
Can we count on you as a resource
What are the risks?

Timeline for the System Indicators Team
Timeline	Task	Completed
July 19, 2002	Retreat Minutes Disseminated to all team members
July 19, 2002	Identify the Division of Research and Statistics expert that will work with this team to accomplish its goals
By August 31, 2002	All LWIB staff will solicit and report back any feedback and input from local board/private sector and partners in regards to system indicators that should be collected and reported
By August 31,2002	NYSDOL will solicit feedback from State Partners and report back to the SIT
July 22, 2002 through September 15, 2002	Subcommittee will meet with R&S representative to draft measures (indicators) and submit to the SIT for their review and comment. All products developed will be available on a secured web site so that all LWIB staff representatives can provide comments/feedback to the SIT.
By October 1, 2002	All feedback on draft must be submitted to the subcommittee for their review and consideration.
By October 22, 2002	Incorporate comments for meeting at NYATEP pre-day. Compile controversial comments for discussion at the Buffalo meeting.
November 14, 2002	Present to the Local Liaison Subcommittee to the SWIB and possibly the full Board.

Attachment 2 - Local Partners’ Table Feedback on System Indicators

Yonkers Partners' Table

Comments Regarding System Indicators

1. What do you feel should be the focus of the State Wide reporting effort?

The focus should be on continuity and standardization across the board. Trying to compare programmatic apples and oranges will not benefit anyone. We should report on measures that are easily understood by the public, presented in terms that are used by the business sector. Talking about "17 performance standards" etc. means nothing to the business community. The report needs to focus on a Return on Investment -- what are we spending for what outcomes? This does not mean we should strictly look at all outcomes according to the lowest cost, but rather on an overall quality ROI. The report should focus on ensuring comprehensive information is provided to the Feds (including success stories) so that educated decisions can be made when funding workforce development initiatives.

2. What indicators promote continuous improvement and return on investment?

Job retention
Wage Gain
Wage Level (what are we expending on ITA's versus the outcomes; i.e. spending $8000 on training for an individual to attain a $7 per hour job is not an impressive ROI)
Standard ROI equation
Employer Usage
Employer Referral

3. What indicators would add value to your organization/customers?

Entered employment rates
Job creation
Job Retention
Customer (job seeker/business) satisfaction
Employer Usage/Referral

4. What are the success indicators for your system?

Reduction in Welfare roles
Expansion of Economic base - especially in the areas where we are expending funds on training)
Reduced unemployment rates
Market penetration

5. Can we count on you as a resource?

Yes

6. What do you perceive as the risks to system reporting?

Risks include failure to clearly communicate standardization to all systems and their partners.
Information being misused for purposes other than continuous improvement.

Town of Hempstead Partners’ Table

Comments Regarding System Indicators

1. What do you feel should be the focus of the State Wide Reporting Effort?

The focus of the statewide reporting effort should be activities, services and outcomes pertaining to “Shared System Services” provided within the One-Stop system using the collective resources of the partners. In this context, Shared System Services includes the core services listed at Sec. 662.240 of the WIA regulations. This data could be augmented by categorical program data that is clearly defined and common to all programs within the system.

2. What indicators promote continuous improvement and return on investment?

a. The indicators that promote continuous improvement are:

§ Level of importance that the customer perceives regarding the system;

§ Customer satisfaction rates (at various stages of system participation) for both employers and jobseekers;

§ Number and types of complaints, compliments and suggestions received from customers and staff, as well as responses;

§ Staff capacity building sessions conducted in response to complaints, compliments and suggestions and the measurement of the impact on data as a result of those sessions;

§ Entered employment rates;

§ Employment retention;

§ Wage gains;

§ Wage replacement:

§ Market share;

§ Customer retention rates.

b. The indicators that promote return on investment are:

§ Entered employment rates,

§ Retention rates;

§ Wage gains;

§ Wage replacement;

§ Cost per customer served;

§ Cost per entered employment;

§ Program ROI rates;

§ System ROI rate.

3. What indicators would add value to your organization/customers?

All of the above. Unemployment Insurance Earnings Data would add value by validating employment data in a universal manner.

4. What are the success indicators for our system?

From a statistical reporting standpoint, all of the above. From a public relations standpoint, brand identity, product recognition, general public and legislative opinions.

5. Can we count on you as a resource?

Yes.

6. What do you perceive as the risks (to the system, your programs, the state, etc.) of system reporting?

The risks of system reporting include the following:

o Sources of data could be inconsistent and unreliable resulting in unfair comparisons;

o The type of data collected and the method of collecting it could skew results;

o Turf battles could arise over the causes of positive or negative outcomes;

o The results could be interpreted to reflect an overall system failure;

o Reports not presented in the proper context will be meaningless.

Cattaraugus-Allegany Partners' Table

Comments Regarding System Indicators

Group # 1

What indicators would add value to your agency/organization?

1. Market penetration – businesses don’t know what seminars, etc. are available to them; particularly small businesses.

2. Market diversity – Need to expand services (i.e. training) for different industries like agriculture, mining, and FIRE (finance, insurance and real estate). The current focus has been on manufacturing and service industries. Need to establish more contact and have a better mix of business customers.

3. Customer satisfaction – data for individual and business customers is already available

What are the risks? What are the cautions?

Possible risks are related with market penetration in that resources could be exhausted quickly by a higher response of businesses.

It was commented that public/community benefit indicators would be onerous to track due to cycle time of customers in different programs who take a longer time to reach a measurable result. May only be appropriate to certain Partners.

Group #2

What indicators would add value to agency/organization?

1. Customer satisfaction – by the type of resources/services and how it benefited the responding customer.

2. Market penetration- both businesses and individuals. Can’t track by WIA enrollments, though, since they are only a small segment of the whole.

3. Public/community benefit – possibly use pre-existing data?

It is difficult to pick indicators applicable to all as each agency/organization has its own values, etc.

What indicators would promote system integration and/or build system capacity?

Included are job placements, GEDs obtained, number of customers who no longer need Public Assistance. These could be counted as a shared successes among Partners.

What are the risks? What are the cautions?

It was commented that there is always more behind what you can measure and that data can be misconstrued. The measurable outcomes may receive more attention than others equally or more important outcomes. Also, funding sources may measure indicators differently. Data is also subject to how it is used and perceived, as some may lack knowledge of what the data represents. It was suggested to provide a narrative section along with any data to clarify what it shows.

Tompkins Partners' Table

Comments Regarding System Indicators

Debbie explained the purpose and draft principles of the Success Indicator Team and sought comments and input from Partners.

Partners agreed with the general focus and principles of the state wide reporting effort and to cooperate with the effort provided that it is within their current capacity or with assistance from their respective state agencies.

Success Indictors that promote continuous improvement, return on investment, and customer satisfaction

· Retention – (30 days, 90 days, 6 months)

· Job growth – wage increase

· Return on Investment

· Jobs created/retained

· System responsiveness

· Jobs filled

· Job placement

· Service continuum – other income supports

· Reduction in public subsidy usage (ex. SSI)

· Repeat usage (business)

· Job seeker reduced recidivism

· Shorter Unemployment Insurance duration

· Satisfaction - Surveys (electronically for businesses, phone call sampling for job seekers)

Risks to system reporting on a statewide level were identified as:

· Concise explanation that is understandable.

· Assurance that the information reported will be used to improve the system.

Jefferson-Lewis Partners' Table

Comments Regarding System Indicators

1. What do you feel should be the focus of the State Wide Reporting effort?

The focus of statewide reporting should be on measures that are easily understood and of value to the general public. Resorting to regulatory, compliance-oriented, statistical analysis style of information will ensure that our customers (and others) "turn us off". If we structure it bureaucratically, we will end up talking to ourselves...

2. What indicators promote continuous improvement and return on investment

Three factors jump out immediately in this area: customer satisfaction, usage, and growth rate.

3. What are the success indicators for our system?

Success indicators that REALLY matter (or should matter) are:

Employers:
market penetration
job placements, identified by type and fill rate for job orders (should also be broken down in this area by general #'s as well as by employment type (sectors)

Job Seekers:
job placements
use of the workforce system
customer satisfaction

4. Can we count on you as a resource?

I think I would like to be considered as a resource, as long as we strive for a product that is value-added for the local system, and not a product that evolves into a compliance exercise or instrument.

5. What do you perceive as the risks (to the system, your programs, the state, etc.) of system reporting?

Our biggest risk is NOT producing products that are accurate, complete, and timely. We also must arrive at a single format for producing data/information in a format that subscribes to the K.I.S.S. method (Keep It Simple Stupid). I think that failure to do any of the things mentioned here will hurt our credibility as a system, erode our support base, and leaves us viewed as a tangled bureaucratic web.

Ontario-Seneca-Wayne-Yates Partners' Table

Comments Regarding System Indicators Team

1. What do you feel should be the focus of the State Wide Reporting effort?

Return on Investment
Penetration Rate with services to employers--what percentage are we really serving?
Customer Satisfaction both Job Seeker and Employer
Workforce Retention--effectiveness of system to measure this
Employer Usage
Job Seeker Usage
Job Orders filled

2. What indicators promote continuous improvement and return on investment?

Customer satisfaction
Need a standard ROI formula--not exactly sure what is best

3. What indicators would add value to your organization/customers?

See #1

4. What are the success indicators for our system?

See #1

5. Can we count on you as a resource?

Yes

6. What do you perceive as the risks (to the system, your programs, the state, etc.) of system reporting?

All partners need to have the same definitions and the same reporting systems; each program's federal performance standards are different

Attachment 3 - SIT Subcommittee Meeting Agenda, August 16, 2002

Agenda

System Indicators Team (SIT)

Subcommittee Meeting

Friday, August 16, 2002
10:00am to 3:00pm
Room 218, Technology Bldg.
Mohawk Valley Community College
Utica, New York

Begin – 10:00am

1. Introductions and Meeting expectations/outcomes (15 minutes)

2. Review and Refine Principles (30 minutes)

3. Identify Three Priority System Indicators from prior work (15 minutes)

4. System Indicator 1 – What does it mean? (60 minutes)

Working Lunch (30 minutes)

5. System Indicator 2 – What does it mean? (60 minutes)

6. System Indicator 3 – What does it mean? (60 minutes)

7. Identify tasks/next steps (30 minutes)

Adjourn – 3:00pm

Attachment 4 - SIT Subcommittee Meeting Notes, August 16, 2002

In attendance: Deb Giordano; Joan Sinclair; Judy Davison; Kevin Price; Alice Savino; Laura Quigley; Tony Joseph; Bill Meehan; Karen Coleman
Agenda attached

Summary

Principles

The Statewide system performance measures will focus on and promote the following:

Both the business and the jobseeker customer
Continuous improvement
Integration
Success indicators instead of failure indicators (highlight successes).
Ensure that the process for gathering data is not onerous on our partners (Sampling will be considered to reduce this possibility)
Local context will be provided where possible (e.g., local economic indicators)

System Indicators

1. Market Penetration Rates

Business Indicator	=	Active Business Customer Base Total Number of Firms
Job Seeker Indicator	=	Total # of Individual Customers Served Local Civilian Labor Force

2. Return on Investment

Business Indicator	=	Total System Costs Active Business Customer Base
Jobseeker Indicator	=	Total System Costs Total # of Individual Customers Served

3. Satisfaction (Repeat Usage) Rate

Business Indicator	=	Active Business Customer Base # of Businesses with Repeat Usage
Jobseeker Indicator		Not recommended

Required Data Elements

1. Total number of Businesses in the Local Area

2. Total number of Business Customers in Local Area (i.e., active businesses)

o Total Number of Businesses that received a service/activity in the last 3 years.

o Total number of businesses that have at least one employee and pay into UI system; OR should we use a different definition (e.g., businesses with at least 10 employees and pay into UI system)

R&S will look into the reporting parameters available to us

3. Total System Costs

How do we calculate system costs? Mandated partners only? What about competitive grants that an area receives to deliver services?

4. Total number of Business Customers (active businesses) with Repeat Usage

o Total number of active businesses in the business customer base that have utilized our services/activities for more than once incident of need during a specific period of time(?).

What are the capabilities of OSOS for reporting this? What qualifies as an activity or service? What time period should be used?

5. Total Number of Job Seeker Customers

o Jobseeker customers are those individuals (unduplicated) that receive core, intensive, and/or training services and have received an employment-related service by any of the mandated partners

What capacity do local areas have for obtaining an unduplicated count?

6. Local Civilian Labor Force

o The local civilian labor force is that which is reported by the NYSDOL, R&S, plus 14 and 15 year olds that have working papers, provided such data is available.

R&S will determine if we are able to obtain data for 14 and 15 year olds. If not, we have agreed to not include them for purposes of this measure.

Meeting Discussion

2. Introductions and Meeting Expectations/Outcomes

Team discussed and provided feedback from some of the partners as to what kind of system indicators would be important to them. Some comments follow:

o Some partners do not want to count anything.

o Business wants these questions answered: how many people were served and how much did it cost?

o There are differences between system indicators and programmatic outcomes -- we should be aware of some crossover.

o Can we take information that we collect from each of the programs and compile it into a system report card?

§ Some partners want benefits; time and money and they need the return.

§ This equates to numbers served but how do we capture all that use the system outside of registration/staff assisted?

o Exec. Committee of Board: Partner agencies are reluctant to report, and fear that if we ask for some additional information they will not have the staff or resources to do it.

§ Try to get the Board to think about the statistics that are important to them.

o LWIBs want to know what is already available at the state level, so that we can access that.

§ If data is available already, then not a problem, but LWIBs do not want more requirements/mandates.

o LWIBs like macro level. The Board and community want to know what has the system done that we can indicate improvements: a lower UI rate; affect to welfare; size and growth of labor force; economic development? In addition to macro would like good cycle time and not tied up in process.

o Something that measures and is responsive to business needs.

o Services to business: define what you are going to count and not count. Some services are being provided but we are not getting credit!!

o Partners are concerned that their performance gets blended with the other partners.

§ OSOS is a problem - if all of the partners were using this the next step would be easier. They don't have a lot of faith in the technical aspect of the system.

Expectations

o System Focus will be on a Statewide level. Therefore, we need to define what is relevant across the state; what commonalities do we see despite differences in local economies?

§ This means we need information that is useful at all levels, and then the information can be aggregated at the state level.

o We need to look for a template for system success measurements: What does this mean to people?

o There are going to be three chapters to the template:

1. State; local community;

2. Local CEO; and

3. Local Workforce Investment Boards (LWIBs).

o We will focus on State, and then there can be unique indicators at the local level that will take into consideration local factors.

Outcomes: We need to develop what constitutes the numerator and denominator for each of the indicators. Then we need to explore how we get the necessary data?

3. Review and Refine Principles:

Principles:

o We are going to focus on system wide indicators on a state level.

o We have as our customers: the business and the jobseeker.

o We want to promote continuous improvement.

o We want to promote integration.

o We want to highlight successes.

o We ensure that this process is not onerous on our partners. Do we need to do a sampling instead of all data being utilized? (Sample surveys etc.)

o We would not generate information without a context of the information (e.g., economic indicators, etc.).

o Look at success indicators instead of failure indicators.

4. Identify Three Priority System Indicators from prior work:

1. Market Penetration

2. Customer Satisfaction

3. Return on Investment

5. Market Penetration System Indicators

Business Market Penetration

Discussion

What do we mean by market penetration?
Is anyone measuring this now and how?

Tompkins -- Business Customer base (business who has accessed our services/activities in the last 3 years) and total number of firms (business with as least one employee and pays into UI).

Sullivan -- Said this would be 10 employees or more and active employers (job fairs)

We can start to look at diversity or what it is want to count in the universe. What data is available? We don't have VESID or other partner numbers: Employers need to be put in the database-and post their business contacts.

Total number of firms in that area - What firms do not want to participate in the public employment system? Should we can count in the denominator those firms that are hiring or doing job development work? And then look at how many we served. New hire data is available. Is this a subset of the first? Are we utilizing resources correctly? When we look at hiring do we exclude those that just use the union halls?

It was decided that building the local penetration rate was a good basic goal. Promoting partners to use enter their businesses into the same database was huge. This promotes integration.

Agreement:
Active Business Customer Base (numerator)
Total # of Firms (denominator)

Active Business Customer Base = Active employers; Activity/service within the last 3 years; non-duplicative services

Total # of Firms = At least one employee (UI Records) or do we want to increase what the number of employees should be?

Note: We would like to develop the philosophy that this is a starting point. We are working towards more specific market penetration measures at a later time. This could be accomplished once we have everyone collecting/integrating. Refinement of data will come later as information is collected on a system-wide basis. We need to point to the principles that this measure will encourage continuous improvement.

In terms of LMAs or information what can we collect? DOL's (DoES) definition of what can be collected is: personal visit; telephone conversation; job listing; job fairs?

We need to ensure that there is no duplication of the customer by multiple partners. Active businesses over what period? What can we reasonably access? The measures should be encouraging the use of a system wide collection of business services etc. Business wants a single point of contact. Therefore, we conclude that there needs to be a single data system. We need to follow the principles that we have established and then work towards the end goal of partners' ability to report on these indicators. We need to ensure that all services: Core service, intensive and training services are done with an account manager.

Agreement: Common database OSOS database or other that all partners are utilizing. We need to agree that integration of information/outcomes (non duplication) is more important than what the data collection system is that the local area is utilizing.

Continuous improvement principle: we need to evaluate the outcome, whether it is good or bad. No doubt there will be ranking/comparisons by local areas, however, we need to stress that being a winner or a loser is not what the primary goal is: its about making program improvement based on the data. A LWIB will look at similar areas to see what can be done to improve their outcomes.

A subset for Market Penetration:
# of Firms (numerator)
# of Firms Hiring (denominator)

Problem - How to collect numerator data from areas with multiple Business/Employer databases? Is this a data collection issue or a definition issue? Integrated outcomes are the key and the desired goal. How do we define what an active business is? Time and services/activities: what constitutes the numerator?

0. R&S will explore.

Jobseeker Market Penetration

Jobseeker Market Penetration Rate =:
Total # of individual customers served
the local civilian labor force, plus 14 and 15 year olds (this is the PA Measure)

Problem - Do we have the number of 14 and 15 that have working papers?

1. R&S will see if we collect this information. If this information is not available than we will not include 14-15 year olds.

6. Return on Investment (ROI) System Indicator

Business ROI

Some of the ideas discussed: resources; number of jobs; savings to public welfare; calculating what benefits is generated into the economy (Customer).

Business: What is the business investment? Is it investment of their time? Is this is a return of our investment or of the business? Can you really measure what we invest of our resources into the business? What is our return? What are we looking for as a return/benefit? Suggestions: Tax savings; UI savings; savings of criminal justice costs; return to the economy (multiple benefit to the local economy); costs to business in providing internal human resource activities; lower the UI rating?

Community Investment (tax dollars invested) and community benefit (tax dollars saved or new tax revenues). Is ROI the same for businesses and jobseekers? Is it ROI or value added to the business? Benefit doesn't always accrue to the business that used it.

What are the Core service levels that every partner must provide? Cooperation; collaboration and integration? Referrals to and from the Center? What agencies are entering businesses and jobseekers activity into the common database? How many referrals does each partner make?

Suggestion - Scrape ROI indicator and instead measure System Integration. Value added/Benefits received? Community Colleges generate funds for enrolling individuals –EPE and FTE. We need to communicate that the benefit that the CC receives by generating these funds are as a result of the referrals/participation within the system. Therefore, there are sometimes benefits that we can identify that the partners might not see as a benefit received from their participation in the system. (R&S will look into this)

Conclusion - Statement about why we do not think that ROI is a reasonable system indicator: We should start with System Integration and benefits received from this integration. What do we look for in system integration that could be measured: common intake for all customers; testing and assessment; job development/placement; one point of contact? Businesses want to see a business broker. Integration allows a single point of contact that packages the partner services. How do we equate or value this? The Team believes that if you do well on these things we are improving the ROI for system participation. How do you measure the success? How long did it take to serve the customer? As we get better at integration- did this take less time; what was the cost of OJT; human resource product (what was the saving to the business?). Integration could mean less referrals but better customer satisfaction and intensive services.

Discussion: Should we look at:
Total system costs
Active business customer base (unduplicated)

Job Seeker ROI:
Total individual customers served (unduplicated)
Total system costs

(Individual customers are those that receive core, intensive and training services and have been served by any of the mandated partners. Obtaining unduplicated in the individual customer count might be a challenge if all the partners do not use the same database.)

These are indicators not measures. This could change each year. It needs to include general economic factors; UI; jobs gained or lost; welfare caseload; poverty; wage rates;

Things to think about: how do you calculate system costs? Adding value added? Possibly using the mandated partners only?

7. Customer Satisfaction System Indicators

Business Satisfaction

Issues/Discussion:

0. If we can decide on how to survey businesses to see how satisfied they are with the services/activities with in the system, would DOL be able to handle?

1. Do all local areas want the State to control this process? State administrated and funded?

2. Since the state is already funding the 3 Customer Satisfaction Survey Questions mandated in WIA could the local area add their own questions to that same process?

3. Should it be done twice a year?

4. Confidentiality is an issue for both the jobseeker and business side.

5. Could this be a Web-based application?

6. Surveys may not be the best way to go. Do customers really tell us what we need to know for continuous improvement in a survey?

7. What other means can be utilized to collect/receive customer satisfaction?

8. The surveys that are returned are usually only those that are willing to respond-those might be only those that have positive feedback (afraid to contribute negative comments). How valuable for continuous improvement would this information be?

9. How are the surveys distributed?

10. System-wide/state-wide, how do we do this meaningfully?

11. On the business side, what is a happy customer?

12. Important info: how did you hear about us? (Marketing efforts)

Conclusion:

Not to use surveys to indicate business satisfaction. Indicate business satisfaction by the number of businesses that are using repeat services; or referred others to the system?

Indicator:

# of businesses that receive more than one service (unduplicated)
# of Active Businesses

Alternate if integrated data is not available: Activity Count

Jobseeker Satisfaction

Focus on the system not the Center

13. It is difficult to obtain a customer satisfaction survey and process that will draw meaningful conclusions.

14. Indicators to see if quality increased: use of the website; use of center; number of partners encouraging the use of the system as quality service.

15. Since most of the data that is collected is irrelevant, we should stay to the market penetration.

16. The only other method or indicator that might be useful would be completion rate. But since completion is not the goal of all of the services provided, this would not be the best count or outcome.

Conclusion - Do not develop a system indicator for Customer Satisfaction for the jobseeker.

Attachment 5 - Comments Received via the Web-Board

From: Susan McNamara

Date: Monday, September 09, 2002

Just some thoughts after the Director's Meeting in Albany.... great presentation and a lot of work obviously went into the draft.

On return on investment measure... it appears to focus on cost per customer... my concern is the natural tendency for people to look at the lowest cost as the best, which won't necessarily be the case for our work.. a lot of cost may be spent for a return of a significantly higher skilled and paying job. Maggie made a good point that the feds are moving in that direction, so the measure would be good to watch... but maybe it could be balanced with one that includes the value of the outcome, be it wage or skill level. Sue Mc

From: Jennifer Mesiano for Peter Crisano, Suffolk County Department of Labor

Date: Monday, September 30, 2002

A few comments of the Aug 16 meeting notes of the System Indicators Team:

1. Do not attempt to create a single data system to be used by businesses and partners. A "one size fits all" approach will not work (for example, OSOS) and will divert valuable resources away for the task without fostering system integration.

2. First find out what businesses want to know about the system. The concept of market penetration is interesting, however presenting such a measure as a simple fraction with huge differences in the numerators and denominators is neither useful nor useable. A measure such as this would require the development of a regression formula that accounts for the weight of numerous variables within the labor force. Will this be useful to system users?

3. Find out what data is already being collected. Do not limit yourself to what NYS is doing. The Feds collect and analyze volumes of data on employers that might be useful. Other sources such as local utilities and regional business organizations might be good sources of data.

4. If additional business or partner data needs to be collected let the local Board/Partnership do it and send to a central point for analysis.

5. Use sampling.

From: Gail Breen

Date: Friday, September 06, 2002

FMS WDB has developed a software package and database (starting with 600 local employers with 10 or more employees)to allow all members of our business services consortium to update or share new information on outreach business including business needs, employment t opportunities, and ways to market all programs and funding opportunities. access is available to consortium members through the FMS web site.

From: Karen Springmeier

Date: Monday, August 05, 2002

Team members may want to take a look at TEGL 7-99 dated March 3, 2000. Although it is specific to WIA Performance standards, it does reference applying WIA Core Measures across partner programs.