Create a solution for updating page layout from a template

[futuresoup]

Use case:

We often need to batch update the formatting and layout of our Markdown pages all at once.

For example:

Adding tags, inserting a photo, adding a new page section, creating new callout boxes and graphics, etc.

This needs to be automated so the page layout changes can easily be applied to hundreds of pages.

I'm guessing for now we can write a simply Python script and decide what to do from there.

[bigolboyyo]

I'm working on this with my fork: https://github.com/bigolboyyo/data-magic-wg

I'll have a method to do this and will provide a full walkthrough. I'll need a few more days to finish flushing it out

[bigolboyyo]

I am creating a content_generator class this will allow us to do things like:

cg = ContentGenerator(api_key, "completions", query_params)

or even set it to "markdown" mode and allow us to pass in individual csv, template, and yml files.

See my progress in app/models/content_generator

class ContentGenerator:
    def __init__(
        self,
        api_key,
        mode="markdown",
        yml_file="prompts.yml",
        csv_file="file_info.csv",
        template_md="template.md",
        output_dir="output",
        completion_params=None,
    ):
        self.api_key = api_key
        self.mode = mode
        self.yml_file = yml_file
        self.csv_file = csv_file
        self.template_md = template_md
        self.output_dir = output_dir
        self.request_count = 0
        self.progress = self.handle_progress("load")

        if self.mode == "markdown":
            self._initialize()
        elif self.mode == "completion" and completion_params:
            self.generate_completion(**completion_params)

    def _initialize(self):
        prompts, file_info, template = stage_content(
            self.yml_file,
            self.csv_file,
            self.template_md,
            self.output_dir,
        )
        ```

[bigolboyyo]

I had a quick 1:1 with Shree and Howard getting them up to date with the progress so far and some general flask and file structure info.

I took Howard's original logic and started modeling it out. I left off at the looping of the output pages and the actual writing.

If we look at the original generate_markdown.py we'll see I left off here.

# Loop over pages and generate output
for page in file_info:
    # Generate output for current page
    progress_made = generate_content(prompts['Progress Made']['prompt'].replace('{Topic}', page['Topic']))
    lessons_learned = generate_content(prompts['Lessons Learned']['prompt'].replace('{Topic}', page['Topic']))
    challenges_ahead = generate_content(prompts['Challenges Ahead']['prompt'].replace('{Topic}', page['Topic']))
    best_path_forward = generate_content(prompts['Best Path Forward']['prompt'].replace('{Topic}', page['Topic']))

    # Populate the template with the generated content
    output = template.format(
        topic=page['Topic'],
        progress_made=progress_made,
        lessons_learned=lessons_learned,
        challenges_ahead=challenges_ahead,
        best_path_forward=best_path_forward
    )

    # Write output to file
    with open(f"output/{page['File Name']}", 'w') as f:
        f.write(output)

The main focuses we need to keep in mind are error handling, accidental overwrites, and general file structure.

• We should be able to create a new file or files.
• Choose an output directory | default = 'output'
• We should be able to edit, replace, existing files
• We need to incorporate the image fetching and make it optional and have the ability to add our own custom images
• finish directory setup so our logic is structured (When we write, edit, read, update our files it should know where to put the output.

We'll need an

output/sectors
output/solutions
..etc

That way let's say we want to write a new page with a custom template. We can pass in that unique template for that page.

Or let's say we want to update a specific template already being used?

Finishing this up will stage us for deployment and allow us to move onto the real meat and bones!

[bigolboyyo]

One more thing to focus on is the formatting for our requests.

Currently we are making 4 requests for each page, instead of creating a format and requesting it all in once

    progress_made = generate_content(prompts['Progress Made']['prompt'].replace('{Topic}', page['Topic']))
    lessons_learned = generate_content(prompts['Lessons Learned']['prompt'].replace('{Topic}', page['Topic']))
    challenges_ahead = generate_content(prompts['Challenges Ahead']['prompt'].replace('{Topic}', page['Topic']))
    best_path_forward = generate_content(prompts['Best Path Forward']['prompt'].replace('{Topic}', page['Topic']))
    ```
    
    The idea being to do something like
    
    compiled_request = {
    progress_made,
    lessons_learned,
   challenges_ahead,
   best_path_forward
   }
   
   and then our method for creating the request can do something along the lines of generate_content(compiled_request)
   
   I quickly pseudo coded that up, but I hopefully this is enough to get us all situated and rolling!

[futuresoup]

@bigolboyyo to confirm that we're solving the use case - will your solution allow simple insertions of Markdown in a section of a page or multiple sections of a series of pages in various ways?

For example, often times an editor will need to do the Wikipedia thing and have "Citation Needed" warnings or "Help us make this page better" etc., likely in the middle of the page for a certain section.

I don't want to over-presume what the templating and batch editing needs are until we run into those scenarios but I'm also trying to think ahead :)

Simple is fine for now, we'll figure out a way to at least get the launch going and leave room for adjustment if there are more complex content management needs ahead

[bigolboyyo]

Yeah so the ability to edit, replace files is what we're looking for. By modeling out our original logic it will allow for us to do this very easily.

"For example, often times an editor will need to do the Wikipedia thing and have "Citation Needed" warnings or "Help us make this page better" etc., likely in the middle of the page for a certain section."

We are just talking about conditional rendering at this point. Our logic for this class will have some sort of "edit" method that will grab "whichever specific file" and insert markdown, replace the whole thing, add the "citations needed" logic etc

[bigolboyyo]

Updated file structure on side branch. Will incorporate with Shree and Howard when the planned modeling is finished.
https://github.com/bigolboyyo/data-magic-wg/tree/format

Let's keep the end file structure in mind when writing the logic for writing/reading/editing to files and directories.

I ended up closing the "codebase structure" issue and combining them here. Because our approach is handling both

[bigolboyyo]

Quick example of what I'm prototyping:

if self.unsplash_access_key:
image_url, user_link = self.fetch_unsplash_image(page["Topic"])
if image_url and user_link:
image_section = f"\n\nPhoto by Unsplash"
else:
image_section = ""
else:
image_section = ""

As I build on this I'll do something along the lines of:

• if(self.unsplash):

fetchUnsplashImags(self, "topic", limit=50)

and than we can handle issues with what @futuresoup mentioned.

else:

update template to say "Help us add an image!"

or

"No data on this yet? Want to get involved? Do X Y Z"

We can add functionality to manually add images, do whatever specific updates, uploads, etc

We also always have the option to generate images with openai as well.

https://platform.openai.com/docs/guides/images