Skip to content

Commit 448b747

Browse files
soopelsoopel
committed
Update stats collection to use encrypted consolidated files
- Modified workflow to decrypt/update/encrypt stats files - Consolidated clones and views data into single encrypted files - Removed dated JSON files in favor of clones.json.enc and views.json.enc - Updated .gitignore to allow .json.enc files - Updated stats/README.md to document new encrypted format
1 parent 57af9f0 commit 448b747

File tree

11 files changed

+140
-499
lines changed

11 files changed

+140
-499
lines changed

‎.github/workflows/collect-repository-stats.yml‎

Lines changed: 111 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,21 +14,125 @@ jobs:
1414
with:
1515
ref: ${{ github.head_ref || github.ref_name }}
1616

17+
- name: Decrypt Existing Stats
18+
env:
19+
STATS_ENCRYPTION_KEY: ${{ secrets.STATS_ENCRYPTION_KEY }}
20+
run: |
21+
# Decrypt existing files if they exist
22+
if [ -f "stats/clones.json.enc" ]; then
23+
echo "Decrypting clones.json..."
24+
openssl enc -aes-256-cbc -d -pbkdf2 -in stats/clones.json.enc -out stats/clones.json -k "$STATS_ENCRYPTION_KEY"
25+
else
26+
echo "No encrypted clones.json found, starting fresh"
27+
echo '{"count": 0, "uniques": 0, "clones": []}' > stats/clones.json
28+
fi
29+
30+
if [ -f "stats/views.json.enc" ]; then
31+
echo "Decrypting views.json..."
32+
openssl enc -aes-256-cbc -d -pbkdf2 -in stats/views.json.enc -out stats/views.json -k "$STATS_ENCRYPTION_KEY"
33+
else
34+
echo "No encrypted views.json found, starting fresh"
35+
echo '{"count": 0, "uniques": 0, "views": []}' > stats/views.json
36+
fi
37+
1738
- name: Collect Traffic Data
1839
env:
1940
GH_TOKEN: ${{ secrets.REPO_STATS_TOKEN}}
2041
run: |
2142
echo "Collecting stats at $(date)"
2243
23-
# Collect clones
24-
curl -v -H "Authorization: token $GH_TOKEN" \
44+
# Collect new clones data
45+
curl -s -H "Authorization: token $GH_TOKEN" \
2546
https://api.github.com/repos/${{ github.repository }}/traffic/clones \
26-
> stats/clones_$(date +%Y%m%d).json
47+
> /tmp/new_clones.json
2748
28-
# Collect views
29-
curl -v -H "Authorization: token $GH_TOKEN" \
49+
# Collect new views data
50+
curl -s -H "Authorization: token $GH_TOKEN" \
3051
https://api.github.com/repos/${{ github.repository }}/traffic/views \
31-
> stats/views_$(date +%Y%m%d).json
52+
> /tmp/new_views.json
53+
54+
# Merge clones data
55+
python3 << 'PYTHON'
56+
import json
57+
from collections import defaultdict
58+
59+
# Read existing consolidated data
60+
try:
61+
with open('stats/clones.json', 'r') as f:
62+
existing = json.load(f)
63+
existing_clones = {entry['timestamp']: entry for entry in existing['clones']}
64+
except FileNotFoundError:
65+
existing_clones = {}
66+
67+
# Read new data
68+
with open('/tmp/new_clones.json', 'r') as f:
69+
new_data = json.load(f)
70+
71+
# Merge - new data overwrites existing for same timestamps
72+
for entry in new_data['clones']:
73+
existing_clones[entry['timestamp']] = entry
74+
75+
# Create consolidated output
76+
clones_array = [existing_clones[ts] for ts in sorted(existing_clones.keys())]
77+
output = {
78+
'count': sum(e['count'] for e in clones_array),
79+
'uniques': max(e['uniques'] for e in clones_array) if clones_array else 0,
80+
'clones': clones_array
81+
}
82+
83+
with open('stats/clones.json', 'w') as f:
84+
json.dump(output, f, indent=2)
85+
86+
print(f"Updated clones.json with {len(clones_array)} daily entries")
87+
PYTHON
88+
89+
# Merge views data
90+
python3 << 'PYTHON'
91+
import json
92+
from collections import defaultdict
93+
94+
# Read existing consolidated data
95+
try:
96+
with open('stats/views.json', 'r') as f:
97+
existing = json.load(f)
98+
existing_views = {entry['timestamp']: entry for entry in existing['views']}
99+
except FileNotFoundError:
100+
existing_views = {}
101+
102+
# Read new data
103+
with open('/tmp/new_views.json', 'r') as f:
104+
new_data = json.load(f)
105+
106+
# Merge - new data overwrites existing for same timestamps
107+
for entry in new_data['views']:
108+
existing_views[entry['timestamp']] = entry
109+
110+
# Create consolidated output
111+
views_array = [existing_views[ts] for ts in sorted(existing_views.keys())]
112+
output = {
113+
'count': sum(e['count'] for e in views_array),
114+
'uniques': max(e['uniques'] for e in views_array) if views_array else 0,
115+
'views': views_array
116+
}
117+
118+
with open('stats/views.json', 'w') as f:
119+
json.dump(output, f, indent=2)
120+
121+
print(f"Updated views.json with {len(views_array)} daily entries")
122+
PYTHON
123+
124+
- name: Encrypt Stats Files
125+
env:
126+
STATS_ENCRYPTION_KEY: ${{ secrets.STATS_ENCRYPTION_KEY }}
127+
run: |
128+
echo "Encrypting stats files..."
129+
openssl enc -aes-256-cbc -pbkdf2 -in stats/clones.json -out stats/clones.json.enc -k "$STATS_ENCRYPTION_KEY"
130+
openssl enc -aes-256-cbc -pbkdf2 -in stats/views.json -out stats/views.json.enc -k "$STATS_ENCRYPTION_KEY"
131+
132+
# Remove unencrypted files
133+
rm stats/clones.json stats/views.json
134+
135+
echo "Encryption complete"
32136
33137
- name: Commit Stats
34138
run: |
@@ -38,7 +142,7 @@ jobs:
38142
echo "Git status before add:"
39143
git status
40144
41-
git add -f stats/*.json
145+
git add -f stats/*.json.enc
42146
43147
echo "Git status after add:"
44148
git status

‎.gitignore‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,5 +109,5 @@ swig
109109
openh264
110110
level-zero-*
111111

112-
# Allow JSON files in stats directory for repository statistics
113-
!stats/*.json
112+
# Allow encrypted JSON files in stats directory for repository statistics
113+
!stats/*.json.enc

‎stats/README.md‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,31 @@
11
# Repository Statistics
22

3-
This directory contains automated collection of GitHub repository traffic statistics.
3+
This directory contains historical repository traffic statistics collected automatically.
44

5-
Files are updated daily by the `collect-repository-stats` workflow.
5+
## File Structure
66

7-
- `clones_YYYYMMDD.json` - Repository clone statistics
8-
- `views_YYYYMMDD.json` - Repository view statistics
7+
### Consolidated Statistics
8+
- `clones.json` - All unique daily clone statistics merged from historical collections
9+
- `views.json` - All unique daily view statistics merged from historical collections
10+
11+
Format (same as GitHub API):
12+
```json
13+
{
14+
"count": 1234,
15+
"uniques": 567,
16+
"clones": [
17+
{
18+
"timestamp": "2025-10-03T00:00:00Z",
19+
"count": 4,
20+
"uniques": 1
21+
}
22+
]
23+
}
24+
```
25+
26+
### Historical Dated Files (Source Data)
27+
Historical dated files follow the pattern:
28+
- `clones_YYYYMMDD.json` - Clone statistics snapshot from specific collection date
29+
- `views_YYYYMMDD.json` - View statistics snapshot from specific collection date
30+
31+
These files contain 15-day rolling window data as provided by GitHub's API on that date.

‎stats/clones.json.enc‎

2.13 KB
Binary file not shown.

‎stats/clones_20251017.json‎

Lines changed: 0 additions & 81 deletions
This file was deleted.

‎stats/clones_20251020.json‎

Lines changed: 0 additions & 81 deletions
This file was deleted.

0 commit comments

Comments
 (0)