188 questions
Issue with Bing Search API Key and Web Scraping in Python
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 28.000000000000004%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESJ%3C/text%3E%3C/svg%3E)
Smith Jack
0
Reputation points
Hi everyone,
I hope you're doing well. I am currently working on a Python script that involves web scraping and sentiment analysis. However, I've encountered an issue that I can't seem to resolve. I would greatly appreciate your help in troubleshooting this problem.
Here's a snippet of the Python code I'm using:
# ... (code before)
class WebScraper:
def __init__(self):
self.api_key = os.environ.get("Key 1")
def scrape(self, source, word):
query = quote_plus(f"{word} site:{source}")
url = f"https://api.bing.microsoft.com/v7.0/search?q={query}"
headers = {"Key 2": self.api_key}
response = requests.get(url, headers=headers)
response.raise_for_status()
return response.json()
# ... (other methods)
if __name__ == "__main__":
words = ["A", "B", "C"]
sources = ["twitter.com", "reddit.com"]
main(words, sources)
The problem I'm facing is that when I run the script, I receive the following error message:
Error scraping data: 401 Client Error: PermissionDenied for url: https://api.bing.microsoft.com
I have a suspicion that the issue might be related to the API Key I'm using. I obtained the API Key from the "Keys and Endpoint" section of the "Bing Search" service. Is this the correct method to obtain the API Key? Or am I missing something?
Just to provide some context, I'm running this script on Debian 11, and there don't seem to be any network issues.
Any insights or suggestions you can provide would be greatly appreciated. Thank you in advance for your help!
Best regards,
Steven
Bing | Bing Search APIs | Bing Web Search API
{count} votes
-
romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator2023-08-23T15:45:28.6666667+00:00 @Smith Jack Try modifying your script to use params parameter and pass in the request rather than the URL? Also, I noticed the header name for key is incorrect. It should be
Ocp-Apim-Subscription-Keyinstead of "Key 2"headers = {"Ocp-Apim-Subscription-Key": self.api_key} search_url = "https://api.bing.microsoft.com/v7.0/search" params = {"q": search_term, "textDecorations": True, "textFormat": "HTML"} response = requests.get(search_url, headers=headers, params=params) response.raise_for_status() search_results = response.json()Full reference of using the REST API through python is also documented here.
-
Smith Jack 0 Reputation points
2023-08-24T09:28:01.2+00:00 Thank you for your reply, I have also refactored the code, but the execution
the following command still reports an error:
root@debian: python your_script_name.py --api_key YOUR_API_KEY tuple' object has no attribute 'api_key'Below is the modified code:
import requests import matplotlib.pyplot as plt from bs4 import BeautifulSoup import re from nltk.sentiment.vader import SentimentIntensityAnalyzer from nltk.tokenize import word_tokenize from nltk.corpus import stopwords import jieba from snownlp import SnowNLP from multiprocessing import Pool import logging import argparse # Set the font to a system-installed font plt.rcParams['font.sans-serif'] = 'WenQuanYi Zen Hei' # Load stopwords def load_stopwords(file_path): with open(file_path, "r", encoding="utf-8") as file: stop_words = set(word.strip() for word in file.readlines()) return stop_words def clean_text(text, stop_words): text = re.sub(r"[^\w\s]", "", text) text = re.sub(r"\s+", " ", text) text = re.sub(r"<[^>]+>", "", text) tokens = jieba.lcut(text) filtered_text = [word for word in tokens if word.lower() not in stop_words] return " ".join(filtered_text) def process_query(args): query, api_key, stop_words = args try: # Set the Bing API subscription key and endpoint headers = {"Ocp-Apim-Subscription-Key": api_key} search_url = "https://api.bing.microsoft.com/v7.0/search" params = {"q": query, "textDecorations": True, "textFormat": "HTML"} response = requests.get(search_url, headers=headers, params=params) response.raise_for_status() search_results = response.json() # Perform sentiment analysis and process search results here # ... return query, score_cn except Exception as e: logging.error(f"Exception occurred while processing query '{query}': {str(e)}") return query, None def main(): parser = argparse.ArgumentParser(description="Sentiment Analysis") parser.add_argument("--queries", nargs="+", default=["Python programming", "Java programming", "C++ programming"], help="Queries to search") parser.add_argument("--api_key", required=True, help="Bing API key") parser.add_argument("--processes", type=int, default=4, help="Number of processes for parallel processing") args = parser.parse_args() logging.basicConfig(filename="log.txt", level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s") stop_words = load_stopwords("stopwords.txt") query_args = [(query, args.api_key, stop_words) for query in args.queries] with Pool(args.processes) as pool: results = pool.map(process_query, query_args) word_scores = {query: score_cn for query, score_cn in results if score_cn is not None} # Plot the chart queries = list(word_scores.keys()) scores = list(word_scores.values()) plt.bar(queries, scores) plt.xlabel("Queries") plt.ylabel("Sentiment Score") plt.title("Sentiment Analysis of Queries") plt.show() if __name__ == "__main__": main() -
romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator
2023-08-24T11:10:39.1233333+00:00 @Smith Jack The script worked when I ran it locally, just created a dummy stopwords.txt file and printed the response. See below:
-
romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator
2023-09-03T16:47:26.96+00:00 @Smith Jack Just curious to check, were you able to move forward with your script to use the Bing API? Thanks!!
Sign in to comment
Sign in to answer