rename folder to prompt_evaluations and update README

prompt_evaluations/06_prompt_foo_code_graded_classification/README.md

@@ -0,0 +1,8 @@
+To get started, set your ANTHROPIC_API_KEY environment variable
+
+Then run:
+```
+promptfoo eval
+```
+
+Afterwards, you can view the results by running `promptfoo view`

prompt_evaluations/06_prompt_foo_code_graded_classification/dataset.csv

@@ -0,0 +1,21 @@
+complaint,__expected
+The app crashes every time I try to upload a photo,contains-all:Software Bug
+My printer isn't recognized by my computer,contains-all:Hardware Malfunction
+I can't figure out how to change my password,contains-all:User Error
+The website is completely down I can't access any pages,contains-all:Service Outage
+It would be great if the app had a dark mode option,contains-all:Feature Request
+The software keeps freezing when I try to save large files,contains-all:Software Bug
+My wireless mouse isn't working even with new batteries,contains-all:Hardware Malfunction
+I accidentally deleted some important files can you help me recover them?,contains-all:User Error
+None of your servers are responding is there an outage?,contains-all:Service Outage
+Could you add a feature to export data in CSV format?,contains-all:Feature Request
+"The app is crashing and my phone is overheating","contains-all:Software Bug,Hardware Malfunction"
+I can't remember my password!,contains-all:User Error
+The new update broke something and the app no longer works for me,contains-all:Software Bug
+"I think I installed something incorrectly now my computer won't start at all","contains-all:User Error,Hardware Malfunction"
+"Your service is down and I urgently need a feature to batch process files","contains-all:Service Outage,Feature Request"
+The graphics card is making weird noises,contains-all:Hardware Malfunction
+My keyboard just totally stopped working out of nowhere,contains-all:Hardware Malfunction
+Whenever I open your app my phone gets really slow,contains-all:Software Bug
+Can you make the interface more user-friendly? I always get lost in the menus,"contains-all:Feature Request,User Error"
+The cloud storage isn't syncing and I can't access my files from other devices,"contains-all:Software Bug,Service Outage"

prompt_evaluations/06_prompt_foo_code_graded_classification/promptfooconfig.yaml

@@ -0,0 +1,11 @@
+description: "Complaint Classification Eval"
+
+prompts:
+  - prompts.py:basic_prompt
+  - prompts.py:improved_prompt
+  
+providers:
+  - "anthropic:messages:claude-3-haiku-20240307"
+
+tests: dataset.csv
+

prompt_evaluations/06_prompt_foo_code_graded_classification/prompts.py

@@ -0,0 +1,60 @@
+def basic_prompt(complaint):
+    return f"""
+    Classify the following customer complaint into one or more of these categories: 
+    Software Bug, Hardware Malfunction, User Error, Feature Request, or Service Outage.
+    Only respond with the classification.
+
+    Complaint: {complaint}
+
+    Classification:
+    """
+
+
+def improved_prompt(complaint):
+    return f"""
+    You are an AI assistant specializing in customer support issue classification. Your task is to analyze customer complaints and categorize them into one or more of the following categories:
+
+    1. Software Bug: Issues related to software not functioning as intended.
+    2. Hardware Malfunction: Problems with physical devices or components.
+    3. User Error: Difficulties arising from user misunderstanding or misuse.
+    4. Feature Request: Suggestions for new functionalities or improvements.
+    5. Service Outage: System-wide issues affecting service availability.
+
+    Important Guidelines:
+    - A complaint may fall into multiple categories. If so, list all that apply but try to prioritize picking a single category when possible.
+
+    Examples:
+    1. Complaint: "The app crashes when I try to save my progress."
+    Classification: Software Bug
+
+    2. Complaint: "My keyboard isn't working after I spilled coffee on it."
+    Classification: Hardware Malfunction
+
+    3. Complaint: "I can't find the login button on your website."
+    Classification: User Error
+
+    4. Complaint: "It would be great if your app had a dark mode."
+    Classification: Feature Request
+
+    5. Complaint: "None of your services are loading for me or my colleagues."
+    Classification: Service Outage
+
+    6. Complaint "Complaint: The app breaks every time I try to change my profile picture"
+    Classification: Software Bug
+
+    7. Complaint "The app is acting buggy on my phone and it seems like your website is down, so I'm completely stuck!"
+    Classification: Software Bug, Service Outage
+
+    8. Complaint: "Your software makes my computer super laggy and awful, I hate it!"
+    Classification: Software Bug
+
+    9. Complaint: "Your dumb app always breaks when I try to do anything with images."
+    Classification: 'Software Bug'
+
+    Now, please classify the following customer complaint:
+
+    <complaint>{complaint}</complaint>
+
+    Only respond with the appropriate categories and nothing else.
+    Classification:
+    """