New Data Provided by NCERT

[gprao08]

@RNarmatha
Recently we have got NCERT new data (starter pack programs), in this regard need to we remove old data and ingest new data or directly ingest new data. Because AP School Education Department want to show latest NCERT data only in dashboard.

[RNarmatha]

Hi @gprao08 ,- Thanks for reaching us .
@bommalla - please look into this and provided the neccessary support

[bommalla]

@gprao08

Please find the below link and start doing the ingestion.if you have questions fell free to raise in discussions forum .

https://cqube.sunbird.org/use-cqube/ingesting-the-data/high-level-understanding-of-ingestion

[gprao08]

@bommalla
Hi Rajesh we have issue while uploading the VSK data (latest data provided by NCERT), after National Programe ingestion no data present all dashboards shows 0%. Could you plz help me out in this issue?

[bommalla]

@gprao08

what was the issue ? Can you please explain me in detail ?

[gprao08]

@bommalla
After ingest latest data (removed old NCERT data) it shows all dashboards empty. Before this I have ingest State Specific data but it is also not showing in dashboard.

[gprao08]

@bommalla
Is there any need to remove old data and go with new data? Now NCERT provided new data.

[bommalla]

@gprao08

which version of cQube are you installed ? If you need to remove the old data from the database. please run the below command in nifi container and re ingest the data again.

sudo docker exec -it nifi_app bash
cd Sunbird-cQube-processing-ms/impl/c-qube/ingest
yarn cli nuke-db

Note: If have ingestion issue please go through the below microsite.

https://cqube.sunbird.org/deployment-process/sdc-deployment

[gprao08]

@bommalla
Ok. We will do the process above and re ingest. Later I will post the update. We are using 5.0.3 version application. Tq.

[gprao08]

@bommalla
Could you plz clarify the below.
We have received data from NCERT the file names differ with the actual files which you stated in Nisththa. Both the files are not same. Shall ask the data as stated in Nisththa and shall I go with the same files. Screen shots attached for your reference,

[bommalla]

@gprao08
Please find the new cQube vsk template and check the New_nishtha tab.based on the template you can check the data which you received from the NCERT.

https://docs.google.com/spreadsheets/d/10yWmivrqtxdRLiJpX_ytzxbvoGG0WNWEI-7RLV8zelM/edit#gid=208908870

[gprao08]

@bommalla
ok. Tq.

[gprao08]

@bommalla
We got the missing files as mentioned above. Now I identified filename mismatch in latest shared template above with postman collection in VSK_Grammer. Those file names in the above mentioned latest template are
NISHTHA_program_implementation_status.zip,
NISHTHA_course-wise_certification.zip,
NISHTHA_medium_of_instructions.zip,
NISHTHA_Program-wise_achieved_certificates.zip,
NISHTHA_participants_per_program.zip

but in the postman collection VSK_Grammer are

diksha_nishtha_consumption-by-course
diksha_nishtha_consumption-by-district
diksha_nishtha_percentage-enrollment-certification
diksha_nishtha_program-started
diksha_nishtha_tot-courses-medium

Can I proceed with the above or shall I need to update my postman collection? If update required could you plz provide the latest postman collection?

[gprao08]

[bommalla]

@gprao08

Please find the latest Postman collection below. Kindly import the latest Postman collections into Postman and begin the ingestion process.

Postman_collections.zip

[gprao08]

@bommalla
After import all postman collection successfully I upload VSK_Grammer and dimension and checked in the database found 5 & 15 rows selected. Later I tried to upload through National Programs API it is showing the below error:

[bommalla]

@gprao08

API endpoint is wrong.please use below endpoint

https://vsk.ap.gov.in/api/ingestion/national_programs

[gprao08]

@bommalla
Thanks. Now 13 files only found in Miniobucket todays date folder. Two files those are

1. NISHTHA_course-wise_certification.zip and
2. vsk_diksha_etb_coverage-status.zip
   not found even after getting "File is being processed" message.
   

[bommalla]

@gprao08
Thank you for the update. Could you please confirm whether the columns, data, and schema of the two files mentioned above align ?

[gprao08]

@bommalla
Found the two files in error folder.

vsk_diksha_etb_qr-coverage_errors.csv
NISHTHA_course-wise_certification_errors.csv

[gprao08]

@bommalla
In the latest template not mentioned state_name & state_code. But is showing error must have the both coloumns.

[bommalla]

@gprao08

Kindly review both error files for any presence of null values in the data. Correct any identified null values and proceed with re-ingesting the data.

[bommalla]

@gprao08

Please add the state_id and state_code columns in NISHTHA_course-wise_certification.csv file and re ingest the data again.

[gprao08]

@bommalla
Done. Tq. Now all files found in Miniobucket.

[bommalla]

@gprao08

Thanks for the update

[gprao08]

@bommalla
After successfully completes "Run_adapters" and "data_moving_local" all dashboards shows 0 counts. Last time also we face the same issue, Could you plz help me to resolve the issue?
Moreover Even I click on any dashboard it will logout, same issue we faced earlier.

[bommalla]

@gprao08

which version of cQube are you installed ?

[gprao08]

@bommalla
5.0.3

[bommalla]

@gprao08

cQube v5.0.3 version there is no "Data_moving_local" processor group.instead of data moving local please schedule "Run Latest Code local" processor group

[bommalla]

@gprao08

Did you schedule the ingest_data processor group ?

[gprao08]

@bommalla
Yes. completed

[gprao08]

@bommalla @RNarmatha
Thanks for your support, Now all Dashboards showing its counts except PM-Poshan. I will cross check the data and update.