Skip to content

fix: enhanced CSV readings upload for new meter values #1613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
johnny603 wants to merge 6 commits into
base: development
Choose a base branch
from
Open

fix: enhanced CSV readings upload for new meter values #1613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3d06acc
added interface types with meter values, also updated default values
johnny603 Apr 22, 2026
4611a82
feat(csv pipeline): honor readings upload overrides with request-firs…
johnny603 Apr 22, 2026
14dbf2e
chore(pr-1613):address maintainer review feedback regarding CSV readi…
johnny603 Apr 24, 2026
75eec8f
chore(pr-1613): address maintainer review feedback for CSV readings u…
johnny603 Apr 27, 2026
25b5393
chore(dev-setup): stabilize local docker startup so oed-web-1 reaches…
johnny603 May 1, 2026
c33dc57
Merge branch 'development' into fix/csvReadings/1519
johnny603 May 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
# * file, You can obtain one at http://mozilla.org/MPL/2.0/.
# *

version: "3.8"
services:
# Database service. It's PostgreSQL, see the
# Dockerfile in ./database
Expand All @@ -21,8 +20,8 @@ services:
interval: 10s
timeout: 10s
retries: 3
# ports:
# - "5432:5432"
ports:
- "5432:5432"
# Uncomment the above lines to enable access to the PostgreSQL server
# from the host machine.
# Web service runs Node
Expand All @@ -33,9 +32,9 @@ services:
- OED_SERVER_PORT=3000
- OED_DB_USER=oed
- OED_DB_DATABASE=oed
- OED_DB_TEST_DATABASE=oed_testing
- OED_DB_TEST_DATABASE=oed
- OED_DB_PASSWORD=opened
- OED_DB_HOST=database # Docker will set this hostname
- OED_DB_HOST=database # Docker service name inside the compose network
- OED_DB_PORT=5432
- OED_TOKEN_SECRET=?
- OED_LOG_FILE=log.txt
Expand Down
153 changes: 140 additions & 13 deletions src/client/app/components/csv/ReadingsCSVUploadComponent.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,13 @@ import { selectIsAdmin } from '../../redux/slices/currentUserSlice';
import { ReadingsCSVUploadPreferences } from '../../types/csvUploadForm';
import { TrueFalseType } from '../../types/items';
import { MeterData, MeterTimeSortType } from '../../types/redux/meters';
import { DisableChecksType } from '../../types/redux/units';
import { submitReadings } from '../../utils/api/UploadCSVApi';
import { ReadingsCSVUploadDefaults } from '../../utils/csvUploadDefaults';
import { showErrorNotification, showSuccessNotification } from '../../utils/notifications';
import { useTranslate } from '../../redux/componentHooks';
import FormFileUploaderComponent from '../FormFileUploaderComponent';
import TimeZoneSelect from '../TimeZoneSelect';
import TooltipHelpComponent from '../TooltipHelpComponent';
import TooltipMarkerComponent from '../TooltipMarkerComponent';
import CreateMeterModalComponent from '../meters/CreateMeterModalComponent';
Expand Down Expand Up @@ -116,7 +118,7 @@ export default function ReadingsCSVUploadComponent() {
}));
};

const handleFileChange = (file: File) => {
const handleFileChange = (file: File | null) => {
Copy link
Copy Markdown
Member

@huss huss Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this directly related to the changes you are making or something you noticed since it checks if file is okay on the next line?

if (file) {
setSelectedFile(file);
if (file.name.slice(-4) === '.csv' || file.name.slice(-3) === '.gz') {
Expand All @@ -132,6 +134,13 @@ export default function ReadingsCSVUploadComponent() {
}
}
};

const handleTimeZoneChange = (timeZone: string) => {
setReadingsData(prevData => ({
...prevData,
timeZone: timeZone || ''
}));
};
/* END of Handlers for each type of input change */

useEffect(() => {
Expand Down Expand Up @@ -672,6 +681,124 @@ export default function ReadingsCSVUploadComponent() {
</Input>
</FormGroup>
</Col>
<Col>
<FormGroup>
<Label>
<div className='pb-1'>
{translate('meter.time.zone')}
</div>
</Label>
<TimeZoneSelect current={readingsData.timeZone ?? null} handleClick={timeZone => handleTimeZoneChange(timeZone)} />
Copy link
Copy Markdown
Member

@huss huss Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asking about the null here and not used with some other TimeZoneSelect uses. I think all these null/''/undefined are probably related.

</FormGroup>
</Col>
</Row>
<Row xs='1' lg='2'>
<Col>
<FormGroup>
<Label for='minVal'>
<div className='pb-1'>
{translate('meter.minVal')}
</div>
</Label>
<Input
type='number'
id='minVal'
name='minVal'
value={readingsData.minVal || ''}
onChange={e => {handleChange(e);}}
/>
</FormGroup>
</Col>
<Col>
<FormGroup>
<Label for='maxVal'>
<div className='pb-1'>
{translate('meter.maxVal')}
</div>
</Label>
<Input
type='number'
id='maxVal'
name='maxVal'
value={readingsData.maxVal || ''}
onChange={e => {handleChange(e);}}
/>
</FormGroup>
</Col>
</Row>
<Row xs='1' lg='2'>
<Col>
<FormGroup>
<Label for='minDate'>
<div className='pb-1'>
{translate('meter.minDate')}
</div>
</Label>
<Input
type='text'
id='minDate'
name='minDate'
value={readingsData.minDate || ''}
onChange={e => {handleChange(e);}}
/>
</FormGroup>
</Col>
<Col>
<FormGroup>
<Label for='maxDate'>
<div className='pb-1'>
{translate('meter.maxDate')}
</div>
</Label>
<Input
type='text'
id='maxDate'
name='maxDate'
value={readingsData.maxDate || ''}
onChange={e => {handleChange(e);}}
/>
</FormGroup>
</Col>
</Row>
<Row xs='1' lg='2'>
<Col>
<FormGroup>
<Label for='maxError'>
<div className='pb-1'>
{translate('meter.maxError')}
</div>
</Label>
<Input
type='number'
id='maxError'
name='maxError'
value={readingsData.maxError || ''}
onChange={e => {handleChange(e);}}
/>
</FormGroup>
</Col>
<Col>
<FormGroup>
<Label for='disableChecks'>
<div className='pb-1'>
{translate('meter.disableChecks')}
</div>
</Label>
<Input
type='select'
id='disableChecks'
name='disableChecks'
value={readingsData.disableChecks || DisableChecksType.reject_all}
onChange={e => {handleChange(e);}}
>
{Object.values(DisableChecksType).map(disableChecks => {
return (
<option value={disableChecks} key={disableChecks}>{translate(`DisableChecksType.${disableChecks}`)}</option>
);
})}
</Input>
</FormGroup>
</Col>
</Row>
{/* TODO This feature is not working perfectly so disabling from web page but allowing in curl.
Rest of changes left so easy to add back in.
Expand All @@ -680,7 +807,7 @@ export default function ReadingsCSVUploadComponent() {
originally added to the web page input of import but decided to not allow
it at this time. Thus, the code was commented out. As of now
there is no plan to make this generally available due to its limitations.*/}
{/*
{/**
<Label check>
<Input
checked={useMeterZone}
Expand All @@ -689,11 +816,11 @@ export default function ReadingsCSVUploadComponent() {
onChange={handleCheckboxChange}
/>
<div className='ps-2'>
{translate('csv.readings.param.use.meter.zone' />
{translate('csv.readings.param.use.meter.zone')}
</div>
</Label>
*/}
{/*
{/**
<Label check>
<Input
checked={warnOnCumulativeReset}
Expand All @@ -702,7 +829,7 @@ export default function ReadingsCSVUploadComponent() {
onChange={handleCheckboxChange}
/>
<div className='ps-2'>
{translate('csv.readings.param.use.meter.zone' />
{translate('csv.readings.param.use.meter.zone')}
</div>
</Label>
*/}
Expand All @@ -720,11 +847,11 @@ export default function ReadingsCSVUploadComponent() {
</div>
<div className='pb-5'>
</div>
</Col>
</Row>
</Form>
</>)}
</Container>
</>
);
}
</Col>
</Row>
</Form>
</>)}
</Container>
</>
);
}
8 changes: 8 additions & 0 deletions src/client/app/types/csvUploadForm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,20 @@
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */

import { MeterTimeSortType } from '../types/redux/meters';
import { DisableChecksType } from './redux/units';

export interface CSVUploadPreferences {
meterIdentifier: string;
gzip: boolean;
headerRow: boolean;
update: boolean;
timeZone?: string;
Copy link
Copy Markdown
Member

@huss huss Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious why these have a ? and the others don't.

minVal?: string;
maxVal?: string;
minDate?: string;
maxDate?: string;
maxError?: string;
disableChecks?: DisableChecksType;
}

export interface ReadingsCSVUploadPreferences extends CSVUploadPreferences {
Expand Down
8 changes: 6 additions & 2 deletions src/client/app/utils/api/UploadCSVApi.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ export const submitReadings = async (uploadPreferences: ReadingsCSVUploadPrefere
warnOnCumulativeReset: uploadPreferences.warnOnCumulativeReset
};
for (const [preference, value] of Object.entries(uploadPreferencesForm)) {
formData.append(preference, value.toString());
if (value !== undefined && value !== null) {
Copy link
Copy Markdown
Member

@huss huss Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would hope that undefined/null are not the value for these but if it was would sending them cause an issue. I wanted your insight into this change.

formData.append(preference, value.toString());
}
}
formData.append('csvfile', readingsFile); // It is important for the server that the file is attached last.

Expand All @@ -61,7 +63,9 @@ export const submitMeters = async (uploadPreferences: MetersCSVUploadPreferences
update: uploadPreferences.update
};
for (const [preference, value] of Object.entries(uploadPreferencesForm)) {
formData.append(preference, value.toString());
if (value !== undefined && value !== null) {
formData.append(preference, value.toString());
}
}
formData.append('csvfile', metersFile); // It is important for the server that the file is attached last.

Expand Down
10 changes: 9 additions & 1 deletion src/client/app/utils/csvUploadDefaults.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */

import { DisableChecksType } from '../types/redux/units';
import { ReadingsCSVUploadPreferences, MetersCSVUploadPreferences } from '../types/csvUploadForm';
import { MeterTimeSortType } from '../types/redux/meters';

Expand All @@ -25,7 +26,14 @@ export const ReadingsCSVUploadDefaults: ReadingsCSVUploadPreferences = {
timeSort: MeterTimeSortType.increasing,
update: false,
useMeterZone: false,
warnOnCumulativeReset: false
warnOnCumulativeReset: false,
Copy link
Copy Markdown
Member

@huss huss Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This did not start with this PR but I wanted to note for a probable issue. This is only used in src/client/app/components/csv/ReadingsCSVUploadComponent.tsx. In the meter component it uses const defaultValues = useAppSelector(selectDefaultCreateMeterValues); and that uses adminPreferences. This is probably a better way given our greater use of Redux. The entire code could be swept to find all of these (meter CSV is below) to think through doing this systematically and how they should be layered.

timeZone: '',
minVal: '',
maxVal: '',
minDate: '',
maxDate: '',
maxError: '',
disableChecks: DisableChecksType.reject_all
};

export const MetersCSVUploadDefaults: MetersCSVUploadPreferences = {
Expand Down
16 changes: 13 additions & 3 deletions src/scripts/installOED.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,20 @@ while test $# -gt 0; do
esac
done

# Load .env if it exists

# Load .env if it exists, but do not overwrite values already provided by the
# runtime environment (for example from Docker Compose).
if [ -f ".env" ]; then
source .env
while IFS='=' read -r name value; do
case "$name" in
''|'#'*)
continue
;;
esac

if [ -z "${!name+x}" ]; then
export "$name=$value"
fi
done < .env
fi

# Skip the install if the node_modules were installed before the package files.
Expand Down
2 changes: 1 addition & 1 deletion src/server/db.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ function swapConnection(newConfig, newConnection) {
if (newConnection !== null) {
connmanager.connection = newConnection;
} else {
connmanager = getDB(connmanager.config);
connmanager.connection = getDB(connmanager.config);
}
}

Expand Down
Loading